>> Narrator: From around the globe. It's theCUBE with digital coverage of Postgres vision 2021 brought to you by EDB. >> Welcome back to Postgres Vision 21. My name is Dave Vellante. We're thrilled to welcome Carl Olofsen to theCUBE. Carl is a research vice president at IDC focused on data management. The long-time database analyst is the technologist and market observer. Carl, good to see you again. >> Thanks Dave. Glad to be here. >> All right. Let's let's get into it. Let's talk about, let's go right to the, to the source the open source database space. You know, how, what changes have you seen over the last couple of years in that marketplace? >> Well, this is a dynamic area and it's continuing to evolve. When we first saw the initial open source products like mysQl and PostgreSQL on the early days they were very limited in terms of functionality. They were espoused largely by sort of true believers. You know, people who said everything should be open source. And we saw that mainly they were being used for what I would call rather prosaic database applications. But as time has gone by they both of these products improve. Now there's one key difference, of course, which is a mySQL is company owned open source. So the IP belongs to Oracle corporation. Whereas PostgreSQL is community open source, which means that the IP belongs to the PostgreSQL community. And that can have a big difference in terms of things like licensing and so forth, which really matters now that we're coming into the cloud space because as open-source products moving into the cloud space the revenue model is based on subscriptions. And of course they are always based on subscription to open source cause you don't charge for the license. So what you charge for its support, but in the cloud what you can do is you can set up a database service, excuse me, a database service and then you charge for that service. And if it's open source or it's not open source that actually doesn't matter to the user. If you see what that I mean because they still are paying a subscription fee for a service and they get the service. The main difference between the two types is that if you're a commercial provider of PostgreSQL like enterprise DB, you don't have control over where it goes and you don't have control over the IP and how people use it in different ways. Whereas Oracle owns mySQL so they have a lot more control and they can do things to it on their own. They don't have to consult the community. Now there's also, non-relational open source including MongoDB. And as you may be aware, MongoDB has changed their license. So that it's not possible for third party to offer Mongo DB as a complete managed database service without paying a license fee to MongoDB for that. And that's because they own the IP too. And we're going to see a lot more of this sort of thing. I have conversations with open source all the time and they are getting a little concerned that it has become possible for somebody to simply take their technology, make a lot of money off that. And no money goes back to the community. No money goes back to the IRS. It's a company it's just stays with the supplier. So I think, you know it'll be interesting to see how all this is over time. >> So you're suggesting that the Postgres model then is, is I guess I'll use the word cleaner. And so that feels like it's a it's a benefit or is it a two-edged sword kind of thing? I mean, you were saying before, you know a company controls the IP so they could do things without having to go to the community. So maybe they can do things faster. But at the other hand like you said, you get handcuffed. You think you're going to be able to get a, you know a managed service, but then all of a sudden you're not and the rules change midstream saying it, am I correct? That Postgres, the model is cleaner for the customer? >> Well, you know, I mean, a lot of my friends who are in the open source community don't even consider company owned open source to be true open source because the IP is controlled by a company, not by a community. >> Dave: Right >> So from that perspective certainly Postgres SQL is considered, I don't know if you want to use the word cleaner or more pure or something along those lines, but also because of that the nature of community open source it can be used in many different ways. And so we see Postgres popping up all over the place sometimes partially and sometimes altogether, in other words, a service, a cloud service, we'll take a piece of Postgres and stick it on top of their own technology and offer it. And the reason they do that is they know there are a lot of developers out there who already know how to code for Postgres. So they are immediately first-class users of the service that they're offering. >> So, talk a little bit more about what you're seeing. You just mentioned a lot of different use cases. That's interesting. I didn't realize that was, that was happening. The, what are you seeing in terms of adoption in let's say the last 18, 24 months specific to Postgres? >> Yeah, we're seeing a fair amount of adoption in especially in the middle market. And of course there is rapid adoption in the tech sector. Now, why would that be? Well it's because they have armies of technologists. Who know how to program this stuff. You know, when you, you know, a lot of them will use PostgreSQL without a contract without a support contract, they'll just support themselves. And they can do that because they have the technicians who are capable of doing it. Most regular businesses can't do that. They don't have the staff so they need that support contract. And so that's where a company like enterpriseDB comes. I mentioned them only because they're the leading supplier Postgres to all their other suppliers. >> I was talking to Josh Burgers, red hat and he was, you know, he had just come off a Cubacon and he was explaining kind of what's happening in that community. Big focus of course on security and the whole, you know, so-called shift left. We were having a good discussion about, you know when does it make sense to use, you know Postgres in a container environment should you use Postgres and Kubernetes and he sort of suggested that things have rapidly evolved. There's still, you know, considerations but what are you seeing in terms of the adoption of microservices architectures containers, generally Kubernetes how has that affected the use of things like postsgres? >> So those are all different things or need to be kind of custody. >> Pick your favorite. >> They're related then. So microservices, the microservice concept is that you take an application break it up into little pieces and each one becomes a microservice that's invoked through an API. And then you have this whole structure API system that you use to drive the application and they run. They typically, they run in containers usually Kubernetes govern containers but the reason you do this and this is basically a efficiency because especially in the cloud, you want only to pay for what you use. So when you're running a microservice based application. Applications have lots of little pieces when something needs to be done, microservice fires up it does the thing that needs to be done. It goes away. You only pay for that fraction of a second that the microservice is running. Whereas in a conventional application you load this big heavyweight application. It does stop. It sets some weights with things and does more stuff and sits and waits for things. And you pay for compute for that entire period. So it's much more cost effective to use a microservices application. The thing is that microservice, the concept of microservices is based on the idea that the code is stateless but database code isn't stateless cause it has its attraction to the database which is the ultimate kind of like stateful environment right? So it's a tricky business. Most database technologies that are claimed to be container-based actually run in containers the way they run in servers. In other words, they're not microservice-based they do run in containers. And the reason they're doing that is for portability so that you can deploy them anywhere and you can move them around. But you know deploying a microservice based database is, well, it's it's a big technical project. I mean, that is hard to do. >> Right and so talk about, I mean again we're talking to Josh it was clear that that Kubernetes has evolved, you know quite rapidly at the same time there were cautions. In other words, he would say I think suggested things like, you know, there were known at one point, there were known, you know flaws and known bugs that ship the code that's been been remediated or moderated in terms of that practice but still there's there's considerations just in terms of the frequency of updates. I think he gave the example of when was the last time you know, JVM got, you know, overhauled. And so what kind of considerations should customers think about when considering them, they want the Kubernetes they want the flexibility and the agility but at the same time, if they're going to put it production, they've got to be careful, right? >> Yeah, I think you need to make sure you're using you're using functions that are well-established, you know you wouldn't want to put something into production that's new. They say, oh, here's a new, here's a new operation. Let's try that. And then, you know, you get in trouble. So you want to deal conservative that way you know, Kubernetes is open-source so and the updates and the testing and all that follows a rather slow formal process, you know from the time that the submission comes in to the time that it goes out, whereas you mentioned JVMs JV, but it was owned by Oracle. And so JVMs are managed like products. Now there's a whole sort of legal thing I don't want to get into it as to whether it's legal. They claim it's not libero third parties to build JVMs without paying a licensing. I don't want to talk about that, but it's based on a very state that has a very stable base, you know whereas this area of Kubernetes and govern containers is still rapidly evolving but this is like any technology, right? I mean, when you, if you're going to commit your enterprise to functions that run on an emerging technology then you are accepting some risk. You know, that there's no question about it. >> So we talked about the cloud earlier and the whole trend toward managed services. I mean, how does that specifically apply to Postgres? You can kind of imagine like a sidecar, a little bit of Postgres mixed in with, you know, other services. So what do you see and what do you, what's your telescope say in terms of the the Postgres adoption cloud? How do you see that progressing? >> I think there's a lot of potential. There's a lot of potential there. I think we are nowhere near the option that it should be able to achieve. I say that because for one thing, even though we analyze the future at IDC, that doesn't mean we actually know the future. So I can't say what its adoption will be but I can say that there's a lot of potential there. There's a tremendous number of Postgres developers out there. So there's a huge potential for adoption. And especially in cloud adoption, the main thing that would help that is independent. And I know that enterpriseDB has one independent a managed cloud service. So I think they do. >> Yeah I think so. >> But you know, why do I say that? I say that because alternatives these days there are some small companies that maybe they'll survive and maybe they won't, but that, you know, do you want to get involved with them or the cloud platform providers, but if you use their Postgres you're locked into that cloud platform. You know, if you use Amazon, go press on RDS, right? You're not, you become quickly locked in because you're starting using all the AWS tools that surround it to build and manage your application. And then you can't move. If you see what I mean. >> Dave: Yeah . >> They have have an RDS labor Aurora, and this is actually one of the things that it's really just a thin layer of Postgres interaction code underneath Aurora is their own product. so that's an even deeper level of commitment. >> So what has to happen for, so obviously cloud, you know, big trend. So the Postgres community then adopts the code base for the cloud. Obviously EDB has, you know hundreds of developers contributing to that, but so what does that mean to be able to run in the cloud? Is that making it cloud native? Is that extensions? Is it, you know, what technically has to occur and what has occurred and how mature is it? >> Well, so smaller user organizations are able to migrate fairly quickly cloud because most of their applications are you know, commercially purchased. They're like factories applications. When they move to the cloud, they get the SAS one and often the SAS equivalent runs on Postgres. So that's just fine. Larger enterprises are a real mess. If you've ever been in a large enterprise data center you know what I'm talking about? It's just, there's just servers and storage everywhere. There's, all these applications, databases connections. They are not moving to the cloud anytime soon. But what they are doing is setting up things like private cloud environments and applying in there. And this is a place where if you're thinking about moving to something like a Postgres you know most of these enterprises use the big commercial databases. Oracle SQLserver DB two and so forth. If you're thinking of moving from that to a a PostgreSQL development say, then the smart thing to do would be first to do all your work in the private cloud where you'd have complete control over the environment. It also makes sense still to have a commercial support contract from a vendor that you trust, because I've said this again, unless you are, you know, Cisco or somebody, you know, some super tech company that's got all the technicians you need to do the work. You really don't want to take on that level of risk. If you see that, I mean. Another advantage to working with a supplier, a support supplier, especially if you have a close, intimate relationship is they will speed your security patches on a regular basis which is really important these days, because data security is as you know, a growing concern all over the place. >> So let's stay on the skillsets for a minute. Where do you see the gaps within enterprises? What kind of expertise you mentioned, you know support contracts, what are the types of things that a customer should look for in terms of the the expertise to apply to supporting Postgres databases? >> Well, obviously you want them to do the basics that any software company does, right? You want them to provide you with regular updates and binary form that you can load and, you know test and run. You want to have the you know, 24 hour hotline you know, telephone support, all that kind of thing. I think it's also important to have a solid ability on the part of the vendor that you're working with to provide you with advice and counseling as you, especially, if you're migrating from another technology, help your people convert from what they were using to what they're going to be using. So those are all aspects that I would look for in a vendor for supporting a product like PostgreSQL. >> When you think about the migration to the cloud, you know of course Amazon talks a lot about cloud migration. They have a lot of tooling associated with that. >> Carl: Right. >> But when you step back and look at it it did to a point earlier, I mean a lot of the hardcore mission, critical stuff isn't going to move it, hasn't moved, but a lot of the fat middle, you know, is, are good candidates for it. >> Carl: Right. >> How do you think about that? And how do you look at that? I mean, obviously Oracle is trying to shove everything into OCI and they're, you know, they're all in because they realized that could make a lot of money doing that. But what do you, what are the sort of parameters that we should think about when considering that kind of migration, moving a legacy database into the cloud? >> Well, it has to be done piecemeal. You're not going to be able to do it all at once. You know, if you have hundreds of applications, you're not just you don't even want to, you know, it's a good time to take you into it. And what you've got running, ask yourself are these applications really serving the business interests today and will they in the future or is this a good time to maybe consider something else? Even if you have a packaged application, there might be one that is more aligned with your future goals. So it's important to do that. Look at your data integration, try to simplify it. You know, most data integration that most companies has done piecemeal project by project. They don't reference each other. So you have this chaos of ETL jobs and transformation rules and things like that that are just, you know, even difficult to manage. Now, just forget about any kind of migration or transformation considerations, just trying to run it now is becoming increasingly difficult. You know, maybe you want to change your strategy for doing data integration. Maybe you want to consolidate you want to put more data in one database. I'm not an advocate of the idea that you can put all application data in one database by the way, we know from bitter experience that doesn't work, but we can be rational about the kinds of databases that we use and how they sit together. >> Well, I mean, you've been following this for a long time and you saw the sort of rise and fall of the big data meme. And you know, this idea that you can shove everything into a single place, have a single version of the truth. It's like, it's just never seemed to happen. >> Carl: Right. >> So, you know, Postgres has been around a long time. It's evolved. I mean, I remember when, you know, VMware's ascendancy and people are like, okay, should I, you know should I virtualize my Postgres database is your, you know similar conversations that we were having earlier about Kubernetes. You've seen the move to the cloud. We're going to have this conversation about the edge at some point in time. So what's your outlook for Postgres, the Postgres community and, you know database market overall? >> Well, I really think the future for database growth is in the cloud. That's what all the data we're looking at and the case that's what our recent surveys indicate. As I said before, the rate of change depends on the size of the enterprise. Smaller advices are moving rapidly, large enterprises much more slowly and cautiously for the very simple reason that it's a very complex proposition. And also in some cases, they're wondering if they can move certain data or will they be violating your some sort of regulatory constraint or contractual issue. So they need to deal with those things too. That's why the private cloud is the perfect place to get started and get technology all lined up storing your data center is still under your control no legal issues there, but you can start, you know converting your applications to micro-service architected applications running in containers. You can start replacing your database servers with ones that can run in a container environment and maybe in the future, maybe hope that in the future, some of those will actually also be able to run as microservices. I don't think it's impossible but it just involves programming the database server in a very different way than we've done in the past. But you do those things. You can do those things under your own control over time in your own dataset. And then you reach a point where you want to take the elements of your application environment and say, what pieces of this, can I move to the cloud without creating disruption and issues regarding things like data egress and latency from cloud to data center and that kind of thing. And prepare for that. And then you're doing the step wise and then you start converting in a stepwise manner. I think ultimately it just makes so much sense to be in the cloud that the cloud vendors have economies of scale. They can deploy large numbers of servers and storage systems to satisfy the needs of large numbers of customers and create, you know great considerable savings. Some of which of course becomes their profit which is what's due to them. And some of that comes back to the users. So that's what I expect. We're going to see. And oh gosh, I would say that starting from about three years from now the larger enterprises start making their move and then you'll really start to see changes in the numbers in terms of cloud and cloud revenue. >> Great stuff, Carl, thank you for that. So any cool research you're working on lately, how you're spending your your work time, anything you want to plug? >> Well, working a lot on just as these questions, you know cloud migration is a hot topic, another which is really sort of off the subject. And what we've been talking about is graph database which I've been doing a fair amount of research into. I think that's going to be really important in the coming years and really, you know working with my colleagues in a project called the future of intelligence which looks at all the different related elements not just database, data integration but artificial intelligence, data communications and so on and so forth and how they come together to create a more intelligent enterprise. And that's a major initiative that I see. It's one of the, we call the future of initiatives. >> Great, Carls, thanks so much for coming back to theCUBE. It's great to have you, man. I appreciate it. >> Well, I enjoyed it. Now I have to do it again sometime. >> All right you got it. All right thank you everybody for watching theCUBEs. Continuous coverage of Postgres vision 21. This is Dave Vellante keep it right there. (upbeat music)

>>from around the globe. It's the Cube presenting Cuban cloud brought to you by Silicon Angle. Hi, I'm stupid, man. And welcome back to the Cube on Cloud. We're talking about developers. And while so many people remember the mean from 2010 of Steve Balmer jumping around on stage development developers and developers, uh, many people know what really important is really important about developers. They probably read the 2013 book called The New King Makers by Stephen O. Grady. And I'm really happy to welcome to the program. Rachel Stevens, who is an industry analyst with Red Monk who was co founded by the aforementioned Stephen O. Grady. Rachel, Great to see you. Thank you so much for joining us. >>Thank you so much for having me. I'm excited to be here. >>Well, I've had the opportunity, Thio read some of what you've done. We've interacted on social media. We've got to talk events back when we used to do those in people. And >>I'm so >>glad that you get to come on the program especially. You were the ones I reached out. When we have this developer track, um, if you could just give our audience a little bit about your background. You know, that developer cred that you have Because as I joke, I've got a closet full of hoodies. But, you know, I'm an infrastructure guy by training I've been learning about, you know, containers and serverless and all this stuff for years. But I'm not myself much of developer. I've touched a thing or two in the years. >>Yeah. So happy to be here. Red Monk has been around since 2002 and have kind of been beating that developer drum ever since then, kind of as the company, The founder, Stephen James, notice that the decision making that developers was really a driver for what was actually ending up in the Enterprise. And as even more true, as cloud came onto, the scene is open source exploded, and I think it's become a lot more of a common view now. But in those early days, it was probably a little bit more of a controversial opinion, but I have been with the firm for coming up on five years now. My work is an industry analyst. We kind of help people understand, bottoms up technology, adoption trends, so that that's where I spend my time focusing is what's getting used in the enterprise. Why, what kind of trends are happening? So, yeah, that's where we all come from. That's the history of Red Monk in 30 seconds. >>Awesome. Rachel, you talk about the enterprise and developers For the longest time. I just said there was this huge gap you talk about. Bottoms up. It's like, well, developers use the tools that they want If they don't have to, they don't pay for anything. And the general I t. And the business sides of the house were like, I don't know, We don't know what those people in the corner we're doing, you know, it's important and things like that. But today it feels like that that's closed a bunch. Where are we? In your estimation, you know, our developers do they have a clear seat at the table? The title we have for this is whether the Enterprise Developer is its enterprise development oxymoron. In 2020 and 2021 >>I think enterprise developers have a lot more practical authority than people give them credit for, especially if you're kind of looking at that old view of the world where everything is driven by a buyer decision or kind of this top down purchasing motion. And we've really seen that authority of what is getting used and why change a lot in the last year. In the last decade, even more of people who are able to choose the tools that meet the job bring in tools, regardless of whether they maybe have that official approval through the right channels because of the convenience of trying to get things up and running. We are asking developers to do so much right now and to go faster and thio shifting things left. And so the things that they are responsible for incorporating into the way they are building APS is growing. And so, as we are asking developers to do more and to do more quickly, um, the tools that they need to do those, um, tasks to get these APS built is that the decision making us fall into them? This is what I need. This is what needs to come in, and so we're seeing. Basically, the tools that enterprise is air using are the tools that developers want to be using, and they kind of just find their way into the enterprise. >>Now I want to key off what you were talking about. Just developers were being asked to do Mawr and Mawr. We've seen these pendulum swings in technology. There was a time where it was like, Well, I'll outsource it because that'll be easier and maybe it'll be less expensive. And number one we found it necessarily. It wasn't necessarily cheaper. And number two, I couldn't make changes, and I didn't understand what was happening. So when when I talked to Enterprises today, absolutely. I need to have skills that's internally. I need to be able to respond to things fast, and therefore I need skills that I need people that can build what they have. What what do you see? What are those skill sets that are so important today? Uh, you know, we've talked so many times over the years is to you know, there's there's the skills gap. We don't have enough data scientists. We don't have enough developers way. We don't have any of these things. So what do we have and where things trending? >>Yeah, it's It's one of those things for developers where they both have probably the most full tool set that we've seen in this industry in terms of things that are available to them. But it's also really hard because it also indicates that there is just this fragmentation at every level of the stack. And there's this explosion of choice and decisions that is happening up and down the stack of how are we going to build things? And so it's really tricky to be a developer these days and that you are making a lot of decisions and you are wiring a lot of things together and you have to be able to navigate a lot of things. E think. One of the things that is interesting here is that we have seen the phrase like Full stack developer really carried a lot of panache, maybe earlier this decade and has kind of fallen away. Just because we've realized that it's impossible for anybody to be ableto spanned this whole broad spectrum of all of the things we're asking people to dio. So we're seeing this explosion of choice, which is meaning that there is a little bit more focused and where developers are trying to actually figure out what is my niche. What is it that I'm supposed to focus on. And so it's really just this balancing of act of trying to see this big picture of how to get this all put together and also have this focused area realizing that you have to specialize at some point. >>Rachel is such a great point there. We've actually seen that Cambrian explosion of developer tools that are out there. If you go to the CFCF landscape and look at everything out there or goto any of your public cloud providers, there's no way that anybody even working for those companies no good portion of the tools that are out there so nobody could be a master of everything. How about from a cloud standpoint, you know, there is the discussion of, you know what do I shift? Left What? You know, Can I just say, Okay, this piece of it, it could be a manage service. I don't need to think about it versus what skills that I need to have in house. What is it that's important. And obviously, you know, a zoo analyst. We know it varies greatly across companies, but you know what? What are some of those top things that we need to make sure that enterprises have skill set and the tools in house that they should understand. And what can they push off to their platform of choice? >>Yeah, I think your comment about managed services is really pressing because one of the trends that we're watching closely, it's just this rise of manage services. And it kind of ties back into the concept you had before about like, what an I team. That's they have, like the Nicholas Carr. I t doesn't matter, and we're pushing this all the way. And then we realized, Oh, we've got to bring that all back. Um, but we also realize that we really want as enterprises want to be spending our time doing differentiated work and wiring together, your entire infrastructure isn't necessarily differentiated for a lot of companies. And so it's trying to find this mix of where can I push my abstraction higher or to find a manage service that can do something for me? And we're seeing that happen in all levels of the stack. And so what we're seeing is this rise of composite APS where we're going to say, Okay, I'm gonna pull in back end AP ice from a whole bunch of tools like twilio or stripe or all zero where algo Leah, all of those things are great tools that I can incorporate into my app. And I can have this great user, um, interface that I can use. And then I don't have to worry quite so much about building it all myself. But I am responsible for wiring at all together. So I think it's that wire together set of interest that is happening for developers as the tool set that they are spending a lot of time with. So we see the manage services being important. Um played an important role in how absent composed, and it's the composition of that APs that is happening internally. >>What one of the one of the regular research items that I see a red monk is you know what languages you know. Where are the trends going? There's been relative stability, but then something's changed. You know, I look at the tools that you mentioned Full stack developer. I talked to a full stack developer a couple of years ago, and he's like like like terror form is my life and I love everything and I've used it forever. And that was 18 months, Andi. I kind of laugh because it's like, OK, I managed. I measure a lot of the technology that I used in the decades. Um, not that await. This came out six months ago and it's kind of mature. And of course, you know, C I C d. Come on. If it's six weeks old, it's probably gone through a lot of generations. So what do you see? Do you have any research that you can share as to looking forward? What are the You know what the skill sets we need? How should we be training our force? What do >>we need to >>be looking at in this kind of next decade of cloud? >>Yeah. So when when you spoke about languages, we dio a semi annual review of language usage as a sign on get hub and in discussion as seen on stack overflow, which we fully recognize is not a perfect representation of how these languages are used in the broader world. But those air data sets that we have access to that are relatively large and open eso just before anyone writes me angry letters that that's not the way that we should be doing it, Um, but one of the things that we've seen over time is that there is a lot of relative stability in those top tier languages in terms of how they are used, and there's some movement at the bottom. But the trends we're seeing where the languages are moving is type safety and having a safer language and the communities that are building upon other communities. So things like, um, we're seeing Scotland that is able to kind of piggyback off of being a jvm based language and having that support from Google. Or we're seeing typescript where it can piggyback off of the breath of deployment of JavaScript, things like that. So those things where were combining together multiple trends that developers are interested in the same time combined with an ecosystem that's already rich and full. And so we're seeing that there's definitely still movement in languages that people are interested in, but also, language on its own is probably pretty stable. So, like as you start to make language choices as a developer, that's not where we're seeing a ton of like turnover language frameworks on the other hand, like if you're a JavaScript developer and all of a sudden there's just explosion of frameworks that you need to choose from, that may be a different story, a lot more turnover there and harder to predict. But language trends are a little bit more stable over >>time, changing over time. You know, Boy, I I got to dig into, you know, relatively Recently I went down like the jam stack. Uh, ecosystem. I've been digging into a serverless for a number of years. What's your take on that? There's certain people. I talked to him. They're like, I don't even need to be a code. Or I could be a marketing person. And I can get things done when I talked to some developers there like a citizen developers. They're not developers. Come on, you know, I really need to be able to do this, so I'll give you your choices, toe. You know, serverless and some of these trends to kind of ext fan. You know who can you know? Code and development. >>Yeah. So for both translate jam stack and serve Ellis, One of the things that we see kind of early in the iteration of a technology is that it is definitely not going to be the right tool for every app. And the number of APS that they approach will fit for will grow as the tool develops. And you add more functionality over time and all of these platforms expand the capability, but definitely not the correct tool choice in every case. That said, we do watch both of those areas with extreme interest in terms of what this next generation of APS can look like and probably will look like in a lot of cases. And I think that it is super interesting to think about who gets to build these APs, because I e. I think one of the things that we probably haven't landed on the right language yet is what that what we should call these people because I don't think anyone associates themselves as a low code person. Like if you're someone from marketing and all of a sudden you can build something technical, that's really cool, and you're excited about that. Nobody else on your team could build. You're not walking around saying I am a low code marketing person like that, that that's that's that's demeaning. Like you're like. No, I'm technical. I'm a technical market, or look what I just did. And if you're someone who codes professionally for a living like and you use a low code tool to get something out the door quickly and >>you don't >>wanna demean and said, Oh, that was I did a low code that just like everybody, is just trying to solve problems. And everybody, um, is trying to figure out how to do things in the most effective way possible and making trade offs all the time. And so I don't think that the language of low code really is anything that resonates with any of the actual users of low code tools. And so I think that's something that we as an industry need toe work on finding the correct language because it doesn't feel like we've landed there yet. >>Yeah, Rachel, what? Want to get your take on just careers for developers now to think about in 2020 everyone is distributed. Lots of conversations about where we work. Can we bring the remote? Many of the developers I talked to already were remote. I had the chance that interview that the head of remote. Forget lab. They're over 1000 people and they're fully remote. So, you know, remote. Absolutely a thing for developers. But if you talk about careers, it is no longer, you know. Oh, hey, here's my CV. It's I'm on git Hub. You can see the code I've done. We haven't talked about open source yet, so give us your take on kind of developers today. Career paths. Andi. Kind of the the online community there. >>Yeah, this could be a whole own conversation. We'll try to figure out my points. Um, so I think one of the things that we are trying to figure out in terms of balance is how much are we expecting people to have done on the side? It's like a side project Hustle versus doing, exclusively getting your job done and not worrying too much about how many green squares you have on your get hub profile. And I think it's a really emotional and fraught discussion and a lot of quarters because it can be exclusionary for people saying that you you need to be spending your time on the side working on this open source project because there are people who have very different life circumstances, like if you're someone who already has kids or you're doing elder care or you are working another job and trying to transition into becoming a developer, it's a lot to ask. These people toe also have a side hustle. That said, it is probably working on open source, having an understanding of how tools are done. Having this, um, this experience and skills that you can point to and contributions you can point Teoh is probably one of the cleaner ways that you can start to move in the industry and break through to the industry because you can show your skills two other employers you can kind of maybe make your way in is a junior developer because you worked on a project and you make those connections. And so it's really still again. It's one of those balancing act things where there's not a perfect answer because there really is to correct sides of this argument. And both of those things are true. At the same time where it's it's hard to figure out what that early career path maybe looks like, or even advancing in a career path If you're already a developer, it's It's tricky. >>Well, I want to get your take on something to you know, I think back to you know, I go back a decade or two I started working with about 20 years ago. Back in the crazy days were just Colonel Daughter Warg and, you know, patches everywhere and lots of different companies trying to figure out what they would be doing on most of the people contributing to the free software before we're calling it open source. Most of the time, it was their side Hustle was the thing they're doing. What was their passion? Project? I've seen some research in the last year or so that says the majority of people that are contributing to open source are doing it for their day job. Obviously, there's a lot of big companies. There's plenty of small companies. When I goto the Linux Foundation shows. I mean, you've got whole companies that are you know, that that's their whole business. So I want to get your take on, you know, you know, governance, you know, contribution from the individual versus companies. You know, there's a lot of change going on there. The public cloud their impact on what's happening open source. What are you seeing there? And you know what's good? What's bad? What do we need to do better as a community? >>Yeah. E think the governance of open source projects is definitely a live conversation that we're having right now about what does this need to look like? What role do companies need to be having and how things are put together is a contribution or leadership position in the name of the individual or the name of the company. Like all of these air live conversations that are ongoing and a lot of communities e think one of the things that is interesting overall, though, is just watching if you're if you're taking a really zoomed out view of what open source looks like where it was at one point, um, deemed a cancer by one of the vendors in the space, and now it is something that is just absolutely an inherent part of most well tech vendors and and users is an important part of how they are building and using software today, like open source is really an integral tool. And what is happening in the enterprise and what's being built in the enterprise. And so I think that it is a natural thing that this conversation is evolving in terms of what is the enterprises role here and how are we supposed to govern for that? And e don't think that we have landed on all the correct answers yet. But I think that just looking at that long view, it makes sense that this is an area where we are spending some time focusing >>So Rachel without giving away state secrets. We know read Monk, you do lots of consulting out there. What advice do you give to the industry? We said we're making progress. There's good things there. But if we say okay, I wanna at 2030 look back and say, Boy, this is wonderful for developers. You know, everything is going good. What things have we done along the way? Where have we made progress? >>Yeah, I think I think it kind of ties back to the earlier discussion we were having around composite APS and thinking about what that developer experience looks like. I think that right now it is incredibly difficult for developers to be wiring everything together and There's just so much for developers to dio to actually get all of these APs from source to production. So when we talk with our customers, a lot of our time is spent thinking, How can you not only solve this individual piece of the puzzle, but how can you figure out how to fit it into this broader picture of what it is the developers air trying to accomplish? How can you think about where your ATF, It's not on your tool or you your project? Whatever it is that you are working on, how does this fit? Not only in terms of your one unique problem space, but where does this problem space fit in the broader landscape? Because I think that's going to be a really key element of what the developer experience looks like in the next decade. Is trying to help people actually get everything wired together in a coherent way. >>Rachel. No shortage of work to do there really appreciate you joining us. Thrilled to have you finally as a cube. Alumni. Thanks so much for joining. >>Thank you for having me. I appreciate it. >>All right. Thank you for joining us. This is the developer content for the cube on cloud, I'm stew minimum, and as always, thank you for watching the Cube.

>> [Narrator} From theCUBE studios in Palo Alto, in Boston, connecting with thought leaders all around the world. This is theCUBE conversation. >> Hi, I'm Stu Miniman and welcome back to theCUBE on cloud. We're talking about developers and well, so many people remember the meme from 2010 of Steve Ballmer jumping around on stage developer, developers and developers. Many people know what is really important about developers they probably read the 2013 book called "The New Kingmakers" by Stephen O'Grady. And I'm really happy to welcome to the program Rachel Stephens who's an industry analyst with RedMonk who was cofounded by the aforementioned Stephen O'Grady. Rachel great to see you. Thank you so much for joining us. >> Well, thank you so much for having me. I'm excited to be here. I've had the opportunity to read some of what you've done. We've interacted on social media. We've come to talk at events back when we used to do those in people. In person I don't- >> Busy times >> So glad that you get to come on the program, especially you were the ones that I reached out when we had this developer track. If you could just give our audience a little bit about your background that developer credit that you have because as I joke, I've got a closet full of hoodies but I'm an infrastructure guy by training. I've been learning about, containers and serverless and all this stuff for years but I'm not myself much a developer I've touched a thing or two in the years. >> Yeah. So happy to be here. RedMonk has been around since 2002 and have kind of been beating that developer drum ever since then kind of. As the company, I'm the founder. Stephen James noticed that the decision making the developers is really a driver for what was actually ending up in the enterprise. And as even more true as cloud came onto the scene as open source exploded. And I think it's become a lot more of a common view now but in those early days, it was probably a little bit more of a controversial opinion. But I have been with the firm for coming up on five years now. I work as an industry analyst. We kind of help people understand bottoms up technology adoption trends. So that that's where I spend my time focusing is what's getting used in the enterprise. Why, what kind of trends are happening? And so, yeah, that's where we all come from. That's the history of RedMonk in 30 seconds. >> Awesome. Rachel, you talk about the enterprise and developers. For the longest time I just said there was this huge gap. You talk about bottoms up. It's like, well, developers use the tools that they want. If they don't have to, they don't pay for anything. And the general IT and the business sides of the house were like, "We don't know what those people in the corner are doing, it's important." And things like that. But today it feels like that that's closed a bunch. Where are we in your estimation? Are our developers, do they have a clear seat at the table? The title we had for this is whether the enterprise developer is enterprise developer and oxymoron in 2020, in 2021? >> I think enterprise developers have a lot more practical authority than people give them credit for, especially if you're kind of looking at that old view of the world where everything is driven by a buyer decision or kind of this top down purchasing motion. And we've really seen that authority of what is getting used and why change a lot in the last year, And like last decade, even more of people who are able to choose the tools that meet the job and bring in tools, regardless of whether they may be have that official approval through the right channels. Because of the convenience of trying to get things up and running we are asking developers to do so much right now and to go faster and shifting things left. And so the things that they are responsible for incorporating into the way they are building apps is growing, and so as we are asking developers to do more and to do more quickly, the tools that they need to do those tasks to get these apps built, the decision making is falling to them. This is what I need. This is what needs to come in. And so we are seeing basically the tools that enterprise are using are the tools that developers want to be using and they kind of just find their way into the enterprise. >> Now, I want to key off what you were talking about. Just developers are being asked to do more and more. We see these pendulum swings in technology. There was a time where it was like, "Well, I'll outsource it because that'll be easier and maybe it'll be less expensive." And number one, we found it wasn't necessarily cheaper. Number two, I couldn't make changes and I didn't understand what was happening. So when I talked to enterprises today absolutely, I need to have skillsets internally. I need to be able to respond to things fast and therefore I need skills and I need people that can build what they have. What do you see? What are those skill sets that are so important today? we've talked so many times over the years there's the skills gap. We don't have enough data scientists. We don't have enough developers. We don't have any of these things. So what do we have? And where were things trending? >> Yeah, it's one of those things for developers where they both have probably the most full tool set that we've seen in this industry in terms of things that are available to them. But it's also really hard because it also indicates that there's just this fragmentation at every level of the stack. And there's this explosion of choice in decisions that is happening up and down the stack of how are we going to build things. And so it's really tricky to be a developer these days in that you are making a lot of decisions, and you are wiring a lot of things together, and you have to be able to navigate a lot of things. And I think one of the things that is interesting here is that we have seen the phrase like full stack developer really carried a lot of panache maybe earlier this decade and has kind of fallen away just because we've realized that it's impossible for anybody to be able to span this whole broad spectrum of all of the things we are asking people to do. So we're seeing this explosion of choice which is meaning that there is a little bit more focus in where developers, we're trying to actually figure out what is my niche, what is it that I'm supposed to focus on? And so it's really just this balancing of act of trying to see this big picture of how to get this all put together and also have this focused area realizing that you have to specialize at some point. >> Rachel is such a great point there we've absolutely seen that Cambrian explosion of developer tools that are out there. If you go to the CNCF as landscape and look at everything out there or go to any of your public cloud providers there's no way that anybody even working for those companies know a good portion of the tools that are out there. So nobody can be a master of everything. How about from a cloud standpoint? There's the discussion of, what do I shift left? Can I just say okay, this piece of it, it can be a managed service, I don't need to think about it versus what skills that I need to have in house? What is it that's important? And obviously, as analysts, we know it varies greatly across companies, but what are some of those top things that we need to make sure that enterprises have the skillset and the tools in house that they should understand and what can they push off to their platform of choice? >> Yeah, I think your comment about managed services is really prescient because one of the trends that we are watching closely it's just this rise of managed services. And it kind of ties back into the concept you had before about like what in NITMSA have like the Nicholas car, IT doesn't matter, and we're pushing this all away. And then we realized, "Oh, we got to bring that all back." But we also realized we really want as enterprises want to be spending our time doing differentiated work and why we're together your entire infrastructure isn't necessarily differentiated for a lot of companies. And so it's trying to find this mix of where can I push my abstraction higher or to find a managed service that can do something for me? And we're seeing that happen in all levels of the stack. And so what we're seeing is this rise of composite apps, where we're going to say, "Okay, I'm going to pull in back end APIs from a whole bunch of tools like Twilio or Stripe or Alsera, or Algolia all of those things are great tools that I can incorporate into my app, and I can have this great user interface that I can use. And then I don't have to worry quite so much about building it all myself but I am responsible for wiring it all together. So I think it's that wired together set of interests that is happening for developers has the tool set that they are spending a lot of time with. So we see the managed services being important playing an important role in how apps are composed. And it's the composition of that app sort of is happening internally. >> One of the regular research items that I see at a RedMonk is, what languages, where are the trends going? There's been some relative stability but then some things change. I look at the tool set, you mentioned full stack developer. I talked to a full stack developer a couple of years ago and he's like, "Like, ah." Like Terraform is my life and I love everything and I've used it forever. And that was 18 months. And I kind of laugh because it's like, okay, I measure a lot of the technologies that I use in the decades, not that, "Oh wait, this came out six months ago and it's kind of mature." And of course, CICD come on, if it's six weeks old it's probably gone through a lot of iterations. So what do you say, do you have any research that you can share as to looking forward? What are the skill sets we need? How should we be training our force? What do we need to be looking at in this kind of next decade of cloud? >> Yeah, so when you spoke about languages we do a semi-annual review of language usage as seen on GitHub and discussion as seen on Stack Overflow which we fully recognize is not a perfect representation of how these languages are used in the broader world but those are data sets that we have access to that are relatively large and open. So just before anyone writes me, angry letters I said that's not the way that we should be doing it (laughs) but one of the things that we've seen over time is that there is a lot of relative stability in those top tier languages in terms of how they are used. And there's some movement at the bottom but the trends we're seeing where the languages are moving is type safety and having a safer language and the communities that are building upon other communities. So things like we're seeing Kotlin, that is able to kind of piggyback off of being a JVM based language and having that support from Google or we're seeing TypeScript where it can piggyback off of the breadth of deployment of JavaScript, things like that. So those things where we're combining together multiple trends that developers are interested in the same time, combined with an ecosystem that's already rich and full. And so we're seeing that there's definitely still movement in languages that people are interested in but also language on its own is probably pretty stable. So as you start to make language choices as a developer that's not where we're seeing a ton of like turnover. Language frameworks on the other hand, like if you're a JavaScript developer and all of a sudden, there's just explosion of frameworks that you need to choose from. That's maybe a different story, a lot more turnover there and harder to predict, but language trends are a little bit more stable over time. >> There's a lot change. Changing over time. Boy, I got to dig into, relatively recently I went down like the JAMStack ecosystem I've been digging into serverless for a number of years. What's your take on that? There's certain people I talked to and they're like, "I don't even need to be a coder. I can be a marketing person, and I can get things done." When I talked to some developers they're like, "Citizen developers, they're not developers, come on. I really need to be able to do this." So I'll give you your choice as to, serverless and some of these trends to kind of expand who can code and develop. >> Yeah, so for both trans like JAMstack and serverless, one of the things that we see kind of early in the iteration of a technology is that it is definitely not going to be the right tool for every app. And the number of apps that they approach will fit for, will grow as the tool develops and that you add more functionality over time. And all of these platforms expand the capability but definitely not the correct tool choice in every case. That said we do watch both of those areas with extreme interest in terms of what this next generation of apps can look like and probably will look like in a lot of cases. And I think that it is super interesting to think about who gets to build these apps, because I think one of the things that we probably haven't landed on the right language yet is what we should call these people because I don't think anyone associates themselves as a low code person, like if you're someone from marketing and all of a sudden you can build something technical that's really cool. And you're excited about that nobody else on your team can build. You're not walking around saying, "I am a low code marketing person" Like that's demeaning. Like I know I'm a technical marketer. Look what I just did. And if you're someone who codes professionally for a living and you use a low code tool to get something out the door quickly and you don't want to demean or say, "oh hi, I did a low code, that in a sec." Everybody is just trying to solve problems. And everybody is trying to figure out how to do things in the most effective way possible and making trade offs all the time. And so I don't think that the language of low code really is anything that resonates with any of the actual users of low code tools. And so I think that's something that we as an industry need to work on finding the correct language because it doesn't feel like we've landed there yet. >> Yeah, quick Rachel, what want to get your take on just careers for developers now to think about in 2020, everyone is distributed lots of conversations about where do we work? Can we bring your remote? Many of the developers I talked to already were remote. I had a chance to interview the head of remote for GitHub there were over a thousand people and they're fully remote. So, remote absolutely a thing for developers. But if you talk about careers it's no longer, "Oh, Hey, here's my CV." It's, "I'm on GitHub. You can see the code I've done." We haven't talked about open source yet. So give us your take on kind of developers today, career paths and kind of the online community there. >> Yeah. Oh, this could be its whole own conversation. (laughs) I'll try to figure it out the, my points. So I think one of the things that we are trying to figure out in terms of balance is how much are we expecting people to have done on the side? It's like a side project hustle versus doing exclusively getting your job done and not worrying too much about how many green squares you have on your GitHub profile. And I think it's a really emotional and fraught discussion in a lot of quarters because it can be exclusionary for people saying that you need to be spending your time on the side, working on this open source project because there are people who have very different life circumstances. Like if you're someone who already has kids or you're doing elder care or you are working another job and trying to transition into becoming a developer, it's a lot to ask these people to also have a side hustle. That said, it is probably working on open source having an understanding of how tools are done, having this experience and skills that you can point to and contributions you can point to, is probably one of the cleaner ways that you can start to move in the industry and break through to the industry because you can show your skills to other employers. You can kind of maybe make your way in as a junior developer because you've worked on a project and you make those connections. And so it's really still, again, it's one of those balancing act things where there's not a perfect answer because there really is two correct sides of this argument. And both of the things are true at the same time where it's it's hard to figure out what that early career path maybe looks like or even advancing in a career path if you're already a developer, it's, it's tricky. >> Well, I want to get your take on something too. I go back a decade or two, when I started working with Linux about 20 years ago back in the crazy days where it was just kind of lot of work and patches everywhere, and lots of different companies trying to figure out what they would be doing. And most of the people contributing to the free software before we even were calling it open source most of the time it was their side hustle. It was the thing they're doing. It was their passion project. I've seen some research in the last year or so that says the majority of people that are contributing to open source are doing it for their day job. Obviously there's lots of big companies. There's plenty of small companies. When I go to the Linux Foundation shows I mean, you've got whole companies that, that's their whole business. So I want to get your take on governance, contribution from the individual versus companies there's a lot of change going on there. Heck the public clouds, their impact on what's happening open source. What are you seeing there? And what's good, what's bad? What do we need to do better as a community? >> Yeah, I think the governance of opensource projects is definitely a live conversation that we're having right now about what does this need to look like? What role do companies need to be having, and how things are put together is a contribution or leadership position in the name of the individual or the name of the company. Like all of these are live conversations that are ongoing in a lot of communities. I think one of the things that is interesting overall though is just watching if you're taking a really zoomed out view of what open source looks like, where it was at one point deemed at cancer by one of the vendors in this space, and now it is something that is just absolutely, an inherent part of most tech vendors and end users is an important part of how they are building and using software today. Like open source is really an integral tool in what is happening in the enterprise and what's being built in the enterprise. And so I think that it is a natural thing that this conversation is evolving in terms of what is the enterprise's role here and how are we supposed to govern for that? And I don't think that we have landed on all the correct answers yet but I think that just looking at that long view it makes sense that this is an area where we are spending some time focusing. >> So Rachel, without giving away state secrets we know RedMonk, you do lots of consulting out there. What advice do you give to the industry? We said, we're making progress. There's good things there. But if we say, okay, I want to at 2030, look back and say, "Boy, this is wonderful for developers, everything's going good." What things have we've done along the way, where have we made progress? >> Yeah, so I think it kind of ties back to the earlier discussion we were having around composite apps and thinking about what that developer experience looks like, I think that right now it is incredibly difficult for developers to be wiring everything together. And there's just so much for developers to do to actually, get all of these apps from source to production. So when we talk with our customers, a lot of our time is spent thinking, how can you not only solve this individual piece of the puzzle, but how can you figure out how to fit it into this broader picture of what it is the developers are trying to accomplish? How can you think about where you're art fits not only your tool or your project whatever it is that you are working on, how does this fit? Not only in terms of your one unique problem space but where does this problem space fit in the broader landscape? Because I think that's going to be a really key element of what the developer experience looks like in the next decade, is trying to help people actually, get everything wired together in a coherent way. >> Rachel, no shortage of work to do there, really appreciate you joining us thrilled to have you finally as a CUBE alumni. Thanks so much for joining. >> Thank you for having me. I appreciate it. >> All right. Thank you for joining us. This is the Developer Content for theCUBE on cloud. I'm Stu Miniman. And as always, thank you for watching theCUBE. (upbeat music)

>> From around the globe, it's The Cube, with digital coverage of Red Hat Summit 2020, brought to you by Red Hat. >> Hi, and welcome back, I'm Stu Miniman, this is The Cube's coverage of the Red Hat 2020, bringing you guests from Red Hat and their partner ecosystem, practitioners, where they are around the globe, bringing to them this digital event, and while we wish we could all be together in person, we'll just be together apart for 2020. Happy to welcome to the program, a longtime Red Hatter, but first time, on The Cube, Rich Sharples, who's the senior director of product management inside Red Hat, Rich, thank you so much for joining us. >> Yeah, thanks for the invitation, great to be here. >> All right, so the topic we're going to talk about today is something you've got a long background of the middleware space. But in, Quarkus so, I personally was not familiar with Quarkus. Obviously we know, god, I believe someone told me once that there's like, 2 million open source projects out there, so I believe I can be forgiven for not having every one of them memorized there, but of course anybody in our community is going to know Java. What a huge impact that has had on the industry. Linux and Java are two of the, you know, major movers of how we, you know, build an, you know, deal with application today, so give us a little bit of a framework as to what Quarkus is, you know, why it was created. >> Yeah, so it's no secret that as organizations and developers move to this kind of new styled cloud native development, developing applications running in containers or in a kind of serverless environment that Java is not necessarily the best fit. Java does many incredible things, it's an amazing field of engineering. But many of the coolest things it does, assumes that it's going to be a long running application, it can do this cool dynamic class loading and dynamic optimization as the application runs. Those things are pretty impressive, but they're also fairly, very heavyweight. And in our kind of ephemeral environments, whether containers or functions of service, you don't have long running applications. And you can't make use of those things, so in a Java environment you pay for those radical features that you don't necessarily get any benefit from them. So, you know, where we're really trying to lay focus is ensure developers to continue to use Quarkus, it's still the, you know, the dominant language for enterprise development. You still get the benefits of these new architectures, so ensuring that Java continues to be you know, performant and efficient in these new you know, constrained environments. >> Okay, excellent, so we're not calling it cloud native Java though, right Rich? But we are bringing, if I heard right, Java for things like containers Kubernetes, I even heard functions as a service so, we're talking to server lists of you know, open shift server lists something that's being talked about this week. So help us understand you know, if Java was long in the tooth. You know, what stays the same, what's different, how have people been managing and you know building applications in this environment, because obviously you know, we've been dealing with containers for a number of years now, so what have they been doing so far and, you know, why is Quarkus different from some of the alternatives that are out there. >> Really, the goal is to introduce those that stayed the same. It's not a different language, it's not a fork. It is Java, you're writing Java applications, essentially in the same way you used to write them. And you may be using Microsoft still functions so slight difference in terms of design, but it's, you know, we want to ensure that you can bring your favorite frameworks and wipers with you as well. When you're accessing databases or message brokers. We want to ensure you can still use those technologies so we're trying to bring the whole ecosystem with us, with Quarkus, so those things can run well, in a you know, container or service environment as well. And that's super important because the real benefit here is any organizations face the choice of I want to develop cloud native, I want to develop functions, but I've got this huge investment in Java in terms of skills and you know, tools and tool trains and I don't want to go learn a new language, just because I need to you know, take advantage of things new environments so we're essentially giving developers their cake and allowing them to eat it. We are trying to provide the best of both worlds. Stick with the language you already know and you know, have lots of experience with, and still be able to get the benefits of running in our containerized environment. >> Okay. what are some of the challenges here, so you know from an infrastructure standpoint. My background is, you know, virtualization broke a lot of pieces and containerization does the same thing. As you mentioned, things you know, spin up really fast and they don't stay on nearly as long. You know, god, you mentioned functions as a service, often we're measuring things in milliseconds, so everything genomes, understand what's up how do I manage it, how do I monitor it all of those pieces so, you know, I understand you're saying we take the skill set and what we know. But, you know, there's got to be some on ramp here and some considerations >> Yes, so, yeah, absolutely so, Red has taken on the ramp and ensuring that this ecosystem moves with us. We do a lot of hard work within Quarkus, so developers don't have to. We do some very, very clever stuff that very few organizations, would be able to do because they don't have the depth of knowledge of the Java virtual machine that we do. We're able to take a lot of things that you'd normally start off once only, like loading classes and you know, building kind of memory data around, all the kind of reading configurations all of the things applications do once and only once. Why do it another time? Why not build that into the component time, you're going to do it once but take it out of your runtime environment completely, so there are many ways where we're having to kind of rethink the way you know, applications run. We have to do a reset on what job was built for this environment of long running applications where, if the application took 10 minutes to load up all the stage area and classes and config, it didn't really matter, because it's not going to run for 36 months. You got to do a resale on those design decisions and think very very differently and given with our deep experience with containers and you know, working on things like native, serverless and on deep, deep roots in Java, we were able to do that and really think differently. So, Quarkus takes a lot of that kind of work away from developers they don't have to think too much about it. And by and large, what they can do is focus on their applications and their micro services and read all of that wiring and optimization for them. And hopefully deliver some you know, real significant improvements both in development productivity, but also the kind of runtime resource utilization as well to really lower costs. >> Okay, and Rich, what's is great that's been really the nirvana when you talk about developers is they don't want to have to think about some of that underlying you know, gobbledygook. That was why you know, the term serverless is so polarizing is because from a developer standpoint I don't think about this but everybody screams, but there are servers and there is networking and there's you know, things underneath that I need to think about. So, what is the underlying assumption here. We talked about you know, containers, Kubernetes, functions as a service, what integration is done there? Does this live across? Is it kind of like, you know, does it sit just just on RHEL and therefore everywhere the RHEL lives it's there? Or, help me understand kind of what that underlying you know, substrate is. >> Yeah, right now our focus is RHEL x86, 'cause that's kind of the dominant platform in a cloud. It is just Java, some have that natural kind of portability and you know, as other architectures become important, we can certainly look at those as well. The reason why the underlying machine architecture is important, is because one of the options you have with Quarkus is actually the ability to compile everything down to a binary executable, right? That may give you some additional footprint reduction and performance enhancements. And also if we compile down to native, we do need to think about the underlying operating system and the architecture. But by and large, as a developer you really don't have to care. Just like to you don't have to care with Java today. You also have the option with Quarkus, to run on conventional JVM, open JDK is our preference and if you can run on open JDK, then you can pretty much run anywhere. Under you know, different reasons for compiling down a native, this is running on a traditional JDK, different optimizations, different trade-offs that you'd like to make. >> All right, so Rich, an open source project here, can you tell us a little bit about you know, who's contributed to this, you know, what general adoption is this, and, you know, where are we with the solution today. Is it today ready for production environments? >> Yeah, it's getting close to production ready, yeah, we'll be making this Germany available and during Summit and many of the components we use are tried and tested, again we're not reinventing everything from the ground up. We leverage things like REHL VM, we leverage open JDK, we leverage all our frameworks and library, the developer that are familiar with, we just have to optimize them for Quarkus, so, yeah, much of this is not brand new technology. The existing technology that has that kind of maturity and tolling support. So yeah, we're confident it's production ready. One of the early stages of the development of Quarkus, was to use some of Red Hats own products as goody picks. Actually, you know, optimize those products for containerized environments by rebuilding them on top of Quarkus and that gave us obviously a lot of insight into the general readiness, yeah, the whole kind of eating around and dog food principle. In terms of the organizations in investing Quarkus, you know, we have this kind of have old addedge, we often use at Red Hat, which is you know, if you want to, if you want to move quickly, go alone. If you want to go far, then go with others. We're at a stage, where we've been developing Quarkus very, very rapidly and that's mostly been a Red Hat effort. We've certainly got some help from the mothership IBM and I expect that to be an increase overtime and we're now in a point where we have a Germany available product coming up and we're ready to really kind of expand the ecosystem. So, we're looking for you, whether you're a framework provider, you've written a framework for Java and you want to have that Quarkus provider, ensure that runs really well and partly the kind of growing ecosystem around Quarkus, we're looking for that, we're for, you know, cloud providers to you know, take this technology and see how it runs in other environments and give us feedback. So, yeah, definitely looking to expand that ecosystem of contributors, so we can really turn this into kind of the facto technology for the cloud. >> So, Richard, stop back for us for a second, you've got a long history with Java. You know, why in 2020 is you know, Java still, I believe it's like number two on the language list there. Why is it so important today and why is moving forward to all of these cloud solutions so important for that ecosystem. >> Yeah, I think it comes down to you know, organizations are faced with a tough choice. That they stick with the language that they know and love, which is Java, the language, the relevant applications for the last decade and not be able to take the best advantage of cloud and native or serverless environment. Whereas if they go and learn a new language, Datalog or No.js and you know, kind of hunt around and trying to see if that has the same kind of ecosystem and support. So, we want give organizations a better choice, which is you can stick with a language you already know and love and you have skills and the resources, yeah, you can still take advantage of these new environments and that's you know, I'm mean, fundaments the problem we're trying to solve for your customers. That twice open source projects are, they live or die, depending on, they really do scratch an itch, you know, fulfill a need with real developments. I'm going to think we've certainly from the adoption and interest we've seen with Quarkus, we really do think we've found a very real problem to solve. >> Yeah, Rich, before we wrap up, I just want to give you the opportunity, you know, how is your teams doing, I think you know, Red Hat's making a real concerted effort to make you know, an appropriate tone for the event this week. Trying to make sure it's not you know, some of the usual glam that we normally expect to see, full on the community all together, but, you know, the community is so important and you know, the network of people that, you know, built not only you know, technologies but also careers and you know, relationships, so, give us a insight as to how your teams doing, everybody in these challenging times. >> I think this is another good example of where open source really does show it's resilience. Open source projects are simply very, very distributed. No open source projects rely on an office being open, so your word distributed team all used to work using distributed tools across the world, different time zones. It's kind of natural for us, so we're kind of plugging on, you know, just as we have them in the task, you have a few more dogs in the background and crying babies and you know, we're all humans, we all tolerate that. We have great support from our leaderships, that's Red Hat and IMB. They're very clear that they've got people and families before revenue and that's good to know. Everybody's you know continuing as they can to you know, ensure that we have you know, great technology out there 'cause like I said there's real demand here that needs to filled and we're going to continue doing that. So, yeah, everybody's kind of holding up pretty well, so, let's just see how long this thing goes but again, I do think it is a valuable kind of lesson on the resilience of distributed teams and open source in particular. So, yeah. >> All right, well thank you for that Rich. Just to bring it on home, as you said, the general availability of Quarkus you know, is in front of us here, really expecting the ecosystem in costumers move. Give us a little bit of what we should be looking at going forward, what are some of the kind of maturity steps and what should we expect to see, through the remainder of 2020. >> Yeah, it's going to be a pretty exciting year, I mean, given the changes we were all going through we are going to try and come meet developers, where they are, which is you know, on their laptops and in front of their computers, so, we're going to do, we're playing through a bunch of you know, kind of very quick webinars, you know, quick bye what it takes, you know, interesting features, we're going to do some virtual hackathons as well, so you can actually get people with time and talk with some experts. We have platform for doing that. So, we're pretty excited, we, you know, again with the incident, we can reach a lot of developers very easily. Actually far more than we could at a live even like Summit, so, we're going to make the best of it and try to get at to as many developers as we can with Quarkus and you know, hopefully they'll repay us by investing a little bit of time into it and giving us some feedback and you know, trying some applications and you know, see how it goes. >> All right and you know, final, final question for your Rich, you know, Quarkus, I have to imagine that the Quark, the subatomic particle, you know, came into the naming there. Is there some connection with that? I guess why the name to the project? >> Yeah, I mean that's pretty much it, you know, the Quarkus you know, kind of. (mumbles) Arguably the smallest fundamental particle. >> And can we find something smaller? >> Well, there potentially is something smaller but that's kind of in the realm of quantum mechanics and physics, which I'm not an expert on, so, but yeah, it's meant to mean small and the us bit, the US bit. I'd like to think there was a really good big meaning around that. The meaning is that we understand, that trying to do any kind of brand leadership or trademark protection on a well know server like Quark, is it possible? So, we had to add something to Quark and Quarkus kind of sounded cool. >> All right, Rich Sharples, pleasure to catch up with you, congrats on the progress for Quarkus, definitely looking forward to watching it's progression in the future. >> Thanks, great talking to you. >> All right, I'm Stu Minneman. Lot's more coverage here at Red Hat Summit 2020. Thank you as always for watching The Cube. (gentle music)

>> Voiceover: Live from Boston, Massachusetts, it's the CUBE. Covering your Red Hat Summit 2019. Brought to you by Red Hat. >> And welcome back to our coverage here on the CUBE Red Hat Summit 2019. We're at the BCEC in Beantown, Boston, Massachusetts playing host this week to some 9000 strong attendees, pack keynotes. Just a great three days of programming here and educational sessions. Stu Miniman and I'm John Walls. We're joined by Mike Piech, who's the VP and general manager of Middleware at Red Hat. Mike, good to see you today. >> Great to be back. >> And Mark Little, VP of engineering Middleware at Red Hat. Mark, Good to see you as well, sir. >> You too. >> Yeah. First of, let's just talk about your ideas at the show here. Been here for a few days. As we've seen on the keynote stage, wide variety of first off, announcements and great case studies, great educational sessions. But your impressions of what's going on and some of the announcements we've heard about this week. >> Well, sure. I mean definitely some very big announcements with RHEL 8 and OpenShift 4. So as Middleware we're a little bit more in sort of gorilla mode here while some of the bigger announcements take a lot of the limelight. But nevertheless those announcements and the advances that they represent are very important for us as Middleware. Particularly OpenShift 4 as sort of the next layer up from OpenShift which the developers sort of touch and feel and live and breathe on a daily basis. We are the immediate beneficiaries of much of the advances in OpenShift and so that's something that, we as the Middleware guys sort of make real for the enterprise application developer. >> I'd say, probably for me, building on that in a way, one of the biggest announcements, one of the biggest surprises is gotta be the first keynote where we had Satya from Microsoft on stage with Jim announcing the collaboration that we're doing. I never believed that would ever happen and that's, that's fantastic. Has a benefit for Middleware as well but just for Red Hat as a whole. Who would've thought it? >> John: Who would have thought it, right? Yeah, we actually just had Marco Bill-Peter on and he was talking about, he's like "Look, we've actually had some of our support people up in Redmond now for a couple of years." And we had Chris Wright on earlier and he says "You know, sometimes we got to these shows and you get the big bang announcement. It's like, well, really we're working incrementally along the way and open source you can watch it. Sure sometimes you get the new chipset or there's a new this or that. But you know, it's very very small things." So in the spirit of that, maybe, you know, give us the updates since last time we got together. What's happening in the Middleware space as you said. If we build up the stack, you know, we got RHEL 8, we got OpenShift 4 and you're sitting on top. >> Yeah. Well one aspect that's an event like this makes clear in almost a reverse sort of way. We put a lot of effort particularly in Mark's team in getting to a much more frequent and more incremental release cycle and style, right. So getting away from sort of big bang releases every year, couple of years, to a much more agile incremental again sort of regime of rolling out functionality. Now, one of the downsides of that is that you don't have these big grand product announcements to make a big deal about in the same way as RHEL just did with 8 for example. So we need to rethink how we sort of (Laughs) >> absence the sort of big .0 releases, you know how we sort of batch up interesting news and roll it out at a large event like this. Now one of the things that we have been working on is our application environment narrative. Right now, the whole idea of the story here is that many people talk about Cloud-Native and about having lot's of different capabilities and services in a cloud environment. And as we've sort of gone through the, particularly the last year or so, it's really become apparent from what our customers tell us and from what we really see as the opportunities in the cloud-native world. The value that we bring is engineering all these pieces together, right? So that it's not simply a list of these disparate, disconnected, independent services but rather Middleware in the world of cloud native re-imagined. It is capabilities that when engineered together in the right way they make for this comprehensive, unified, cohesive environment within which our customers can develop applications and run those applications. And for the developer, you get developer productivity and then at runtime, you're getting operational reliability. So there really is a sort of a dual-sided value proposition there. And this notion of Middleware engineered together for the cloud is what the application environment idea is all about. >> Yeah. I'd add kinda one of the things that ties into that which has been big for us at least at summit this year is an effort that we kicked off or we announced two months ago called Quakers and as you all know a lot of what we do within Middleware, within Red Hat is based on Java and Java is still the dominant language in the enterprise but it's been around for 20 years. It developed in a pre-cloud era and that made lots of assumptions on the way in which the Java language and the JVM on which it runs would develop which aren't necessarily that conducive for running, in a cloud environment, a hybrid cloud environment and certainly public cloud environment based on Linux containers and Kubernetes. So, we've been working for a number of years in the upstream open JDK community to try and make Java much more cloud-native itself. And Quakers kind of builds on that. It essentially is what we call a kub-native approach where we optimize all of the Middleware stack upfront to work really really well in Kubernetes and specifically on OpenShift. And it's all Java though, that's the important thing. And now if people look into this they'll find that we're showing performance figures and memory utilization that is on a per with some of the newer languages like Go for instance, very very fast. Typically your boot time has gone from seconds to tens of milliseconds. And people who have seen it demonstrated have literally been blown away cause it allows them to leverage the skills that they've had invested in their employees to learn Java and move to the cloud without telling them "You guys are gonna have to learn a completely new language and start from scratch" >> All right, so Mark, if I get it right cause we've been at the Kubernetes show for a bunch of years but this is, you're looking at kinda the application side of what's happening in those Kubernetes environment >> Mark: Yeah. So many times we've talked about the platforms and the infrastructure down but it's the the art piece on top. Super important. I know down the DevZone people were buzzing around all the Quaker stuff. What else for people that are you know, looking at that kinda cloud-native containerization space? What other areas that they should be looking at when it comes to your space? >> Well, again, tying into the up environment thing, hopefully, you know, you'll have heard of knative and Istio. So knative is, to put it in a quick sentence is essentially an enabler for serverless if you like. It's where we're spinning containers really really quickly based on events. But really any serverless platform lives and dies based on the services in which your business logic can then rely upon. Do I have a messaging service there? Do I have a transaction service or a database service? So, we've been working with, with Google on knative and with Microsoft on knative to ensure that we have a really good story in OpenShift but tying it into our Middleware suite as well. So, many of our Middleware products are now knative enabled if you like. The second thing is, as I mentioned, Istio which is a sidecar approach. I won't go into details on that but again Istio the aim behind that is to remove from the application developer some of the non-functional business logic that they had to put in there like "How do I use a messaging service? How do I secure this endpoint and push it down the infrastructure?" So the security servers, the messaging servers, the cashing servers et cetera. They move out of the business logic and they move into Istio. But from our point of view, it's our security servers that we've been working on for years, it's our transactional servers that we've been working on for years. So, these are bullet-proof implementations that we have just made more cloud-native by embedding them in a way in Istio and like I said, enabling them with knative. >> I think we'd mentioned that Chris Wright was on earlier and one of the things he talked about was, this new data-eccentric focus and how, that's at the core so much of what enterprise is doing these days. The fact that whenever speed is distributed, they are and you've got so many data inputs come in from, so to a unified user trying to get their data the way they wanna see it. You might want it for a totally other reason, right? I'm just curious, how does that influence or how has that influenced your work in terms of making sure that transport goes smoothly? Because you do have so much more to work with in a much more complex environment for multiple uses that are unique, right? >> (Mike) Yeah. >> It's not all the same. >> Huge, huge impact for sure. The whole idea of decomposing an application into a much larger number of much smaller pieces than was done in the past has many benefits probably one of the most significant being the ability to make small changes, small incremental changes and afford a much more trial and error approach to innovation versus more macro-level planning waterfall as they call it. But one of the implications of that is now you have a large number of entities. Whether they be big or small, there's a large number of them running within the estate. And there's the orchestration of them and the interconnection of them for sure but it's a n-squared relationship, right. The more these entities you have, the more potential connections between each of them you have to somehow structure and manage and ensure are being done securely and so on. So that has really driven the need for new ways of tying things together, new ways essentially of integration. It has definitely amplified the need for disciplines, EPI management for example. It has driven a lot of increase demand for an event-driven approach where you're streaming in realtime and distributing events to many receivers and dealing with things asynchronously and not depending on round-trip times for everything to be consistent and so on. So, there's just a myriad of implications there that are very detailed technical-level drive some of the things that we're doing now. >> Yeah, I'll just add that in terms of data itself, you've probably heard this a number of times, data is king. Everything we do is based on data in one way or another, So we as Red Hat as a whole and Middleware specifically, we've had a very strong data strategy for a long time. Just as you've got myriad types of data, you can't assume that one way of storing that data is gonna be right for every type of data that you've got. So, we've worked through the integration efforts on ensuring that no sequel data stores, relational data stores^, in-memory data caching and even the messaging services as a whole is a way of sto^ring data in transit, that allows you to, in some ways it allows you to actually look at it in an event-driven way and make intelligent decisions. So that's a key part of what anybody should do if they are in the enterprise space. That's certainly what we're doing because at the end of the day people are building these apps to use that data. >> Well, gentlemen, I know you have another engagement. We're gonna cut you loose but I do wanna say you're the first guests to get applause. (guests laugh) >> From across all the way there. People at home can't hear but, so congratulations. You've been well received already. >> I think they're clearly tuned in to the renaissance of the job in here. >> Yes. >> Thank you both. >> Thanks for the time. >> Mark: Thanks so much. >> We appreciate that. Back with more, we are watching a Red Hat summer 2019 coverage live on the CUBE. (Upbeat music)

>> Welcome everybody, we're at the Flink Forward conference in San Francisco, at the Kabuki Hotel. Flink Forward U.S. is the first U.S. user conference for the Flink community sponsored by data Artisans, the creators of Flink, and we're here with special guest Kenneth Knowles-- >> Hi. >> Who works for Google and who heads up the Apache Beam Team where, just to set context, Beam is the API Or STK on which developers can build stream processing apps that can be supported by Google's Dataflow, Apache Flink, Spark, Apex, among other future products that'll come along. Ken, why don't you tell us, what was the genesis of Beam, and why did Google open up sort of the API to it. >> So, I can speak as an Apache Beam Team PMC member, that the genesis came from a combined code donation to Apache from Google Cloud Dataflow STK and there was also already written by data Artisans a Flink runner for that, which already included some portability hooks, and then there was also a runner for Spark that was written by some folks at PayPal. And so, sort of those three efforts pointed out that it was a good time to have a unified model for these DAG-based computational... I guess it's a DAG-based computational model. >> Okay, so I want to pause you for a moment. >> Yeah. >> And generally, we try to avoid being rude and cutting off our guests but, in this case, help us understand what a DAG is, and why it's so important. >> Okay, so a DAG is a directed acyclic graph, and, in some sense, if you draw a boxes and arrows diagram of your computation where you say "I read some data from here," and it goes through some filters and then I do a join and then I write it somewhere. These all end up looking what they call the DAG just because of the fact that it is the structure, and all computation sort of can be modeled this way, and in particular, these massively parallel computations profit a lot from being modeled this way as opposed to MapReduce because the fact that you have access to the entire DAG means you can perform transformations and optimizations and you have more opportunities for executing it in different ways. >> Oh, in other words, because you can see the big picture you can find, like, the shortest path as opposed to I've got to do this step, I've got to do this step and this step. >> Yeah, it's exactly like that, you're not constrained to sort of, the person writing the program knows what it is that they want to compute, and then, you know, you have very smart people writing the optimizer and the execution engine. So it may execute an entirely different way, so for example, if you're doing a summation, right, rather than shuffling all your data to one place and summing there, maybe you do some partial summations, and then you just shuffle accumulators to one place, and finish the summation, right? >> Okay, now let me bump you up a couple levels >> Yeah. >> And tell us, so, MapReduce was a trees within the forest approach, you know, lots of seeing just what's a couple feet ahead of you. And now we have the big picture that allows you to find the best path, perhaps, one way of saying it. Tell us though, with Google or with others who are using Beam-compatible applications, what new class of solutions can they build that you wouldn't have done with MapReduce before? >> Well, I guess there's... There's two main aspects to Beam that I would emphasize, there's the portability, so you can write this application without having to commit to which backend you're going to run it on. And there's... There's also the unification of streaming and batch which is not present in a number of backends, and Beam as this layer sort of makes it very easy to use sort of batch-style computation and streaming-style computation in the same pipeline. And actually I said there was two things, the third thing that actually really opens things up is that Beam is not just a portability layer across backends, it's also a portability layer across languages, so, something that really only has preliminary support on a lot of systems is Python, so, for example, Beam has a Python STK where you write a DAG description of your computation in Python, and via Beam's portability API's, one of these sort of usually Java-centric engines would be able to run that Python pipeline. >> Okay, so-- >> So, did I answer your question? >> Yes, yes, but let's go one level deeper, which is, if MapReduce, if its sweet spot was web crawl indexing in batch mode, what are some of the things that are now possible with a Beam-style platform that supports Beam, you know, underneath it, that can do this direct acyclic graph processing? >> I guess what I, I'm still learning all the different things that you can do with this style of computation, and the truth is it's just extremely general, right? You can set up a DAG, and there's a lot of talks here at Flink Forward about using a stream processor to do high frequency trading or fraud detection. And those are completely different even though they're in the same model of computation as, you know, you would still use it for things like crawling the web and doing PageRank over. Actually, at the moment we don't have iterative computations so we wouldn't do PageRank today. >> So, is it considered a complete replacement, and then new used cases for older style frameworks like MapReduce, or is it a complement for things where you want to do more with data in motion or lower latency? >> It is absolutely intended as a full replacement for MapReduce, yes, like, if you're thinking about writing a MapReduce pipeline, instead you should write a Beam pipeline, and then you should benchmark it on different Beam backends, right? >> And, so, working with Spark, working with Flink, how are they, in terms of implementing the full richness of the Beam-interface relative to the Google product Dataflow, from which I assumed Beam was derived? >> So, all of the different backends exist in sort of different states as far as implementing the full model. One thing I really want to emphasize is that Beam is not trying to take the intersection on all of these, right? And I think that your question already shows that you know this, we keep sort of a matrix on our website where we say, "Okay there's all these different "features you might want, "and then there's all these backends "you might want to run it on," and it's sort of there's can you do it, can you do it sometimes, and notes about that, we want this whole matrix to be, yes, you can use all of the model on Flink, all of it on Spark, all of it on Google Cloud Dataflow, but so they all have some gaps and I guess, yeah, we're really welcoming contributors in that space. >> So, for someone whose been around for a long time, you might think of it as an ODBC driver, where the capabilities of the databases behind it are different, and so the drivers can only support some subset of a full capability. >> Yeah, I think that there's, so, I'm not familiar enough with ODBC to say absolutely yes, absolutely no, but yes, it's that sort of a thing, it's like the JVM has many languages on it and ODBC provides this generic database abstraction. >> Is Google's goal with Beam API to make it so that customers demand a level of portability that goes not just for the on-prim products but for products that are in other public clouds, and sort of pry open the API lock in? >> So, I can't say what Google's goals are, but I can certainly say that Beam's goals are that nobody's going to be locked into a particular backend. >> Okay. >> I mean, I can't even say what Beam's goals are, sorry, those are my goals, I can speak for myself. >> Is Beam seeing so far adoption by the sort of big consumer internet companies, or has it started to spread to mainstream enterprises, or is still a little immature? >> I think Beam's still a little bit less mature than that, we're heading into our first stable release, so, we began incubating it as an Apache project about a year ago, and then, around the beginning of the new year, actually right at the end of 2016, we graduated to be an Apache top level project, so right now we're sort of on the road from we've become a top level project, we're seeing contributions ramp up dramatically, and we're aiming for a stable release as soon as possible, our next release we expect to be a stable API that we would encourage users and enterprises to adopt I think. >> Okay, and that's when we would see it in production form on the Google Cloud platform? >> Well, so the thing is that the code and the backends behind it are all very mature, but, right now, we're still sort of like, I don't know how to say it, we're polishing the edges, right, it's still got a lot of rough edges and you might encounter them if you're trying it out right now and things might change out from under you before we make our stable release. >> Understood. >> Yep. All right. Kenneth, thank you for joining us, and for the update on the Beam project and we'll be looking for that and seeing its progress over the next few months. >> Great. Thanks for having me. >> With that, I'm George Gilbert, I'm with Kenneth Knowles, we're at the dataArtisan's Flink Forward user conference in San Francisco at the Kabuki Hotel and we'll be back after a few minutes.

>> Announcer: Big Data Silicon Valley 2017. >> Hey, welcome back, everybody, Jeff Frick here with The Cube. We are live at the historic Pagoda Lounge in San Jose for Big Data SV, which is associated with Strathead Dupe World, across the street, as well as Big Data week, so everything big data is happening in San Jose, we're happy to be here, love the new venue, if you're around, stop by, back of the Fairmount, Pagoda Lounge. We're excited to be joined in this next segment by, who's now become a regular, any time we're at a Big Data event, a Spark event, Holden always stops by. Holden Karau, she's the principal software engineer at IBM. Holden, great to see you. >> Thank you, it's wonderful to be back yet again. >> Absolutely, so the big data meme just keeps rolling, Google Cloud Next was last week, a lot of talk about AI and ML and of course you're very involved in Spark, so what are you excited about these days? What are you, I'm sure you've got a couple presentations going on across the street. >> Yeah, so my two presentations this week, oh wow, I should remember them. So the one that I'm doing today is with my co-worker Seth Hendrickson, also at IBM, and we're going to be focused on how to use structured streaming for machine learning. And sort of, I think that's really interesting, because streaming machine learning is something a lot of people seem to want to do but aren't yet doing in production, so it's always fun to talk to people before they've built their systems. And then tomorrow I'm going to be talking with Joey on how to debug Spark, which is something that I, you know, a lot of people ask questions about, but I tend to not talk about, because it tends to scare people away, and so I try to keep the happy going. >> Jeff: Bugs are never fun. >> No, no, never fun. >> Just picking up on that structured streaming and machine learning, so there's this issue of, as we move more and more towards the industrial internet of things, like having to process events as they come in, make a decision. How, there's a range of latency that's required. Where does structured streaming and ML fit today, and where might that go? >> So structured streaming for today, latency wise, is probably not something I would use for something like that right now. It's in the like sub second range. Which is nice, but it's not what you want for like live serving of decisions for your car, right? That's just not going to be feasible. But I think it certainly has the potential to get a lot faster. We've seen a lot of renewed interest in ML liblocal, which is really about making it so that we can take the models that we've trained in Spark and really push them out to the edge and sort of serve them in the edge, and apply our models on end devices. So I'm really excited about where that's going. To be fair, part of my excitement is someone else is doing that work, so I'm very excited that they're doing this work for me. >> Let me clarify on that, just to make sure I understand. So there's a lot of overhead in Spark, because it runs on a cluster, because you have an optimizer, because you have the high availability or the resilience, and so you're saying we can preserve the predict and maybe serve part and carve out all the other overhead for running in a very small environment. >> Right, yeah. So I think for a lot of these IOT devices and stuff like that it actually makes a lot more sense to do the predictions on the device itself, right. These models generally are megabytes in size, and we don't need a cluster to do predictions on these models, right. We really need the cluster to train them, but I think for a lot of cases, pushing the prediction out to the edge node is actually a pretty reasonable use case. And so I'm really excited that we've got some work going on there. >> Taking that one step further, we've talked to a bunch of people, both like at GE, and at their Minds and Machines show, and IBM's Genius of Things, where you want to be able to train the models up in the cloud where you're getting data from all the different devices and then push the retrained model out to the edge. Can that happen in Spark, or do we have to have something else orchestrating all that? >> So actually pushing the model out isn't something that I would do in Spark itself, I think that's better served by other tools. Spark is not really well suited to large amounts of internet traffic, right. But it's really well suited to the training, and I think with ML liblocal it'll essentially, we'll be able to provide both sides of it, and the copy part will be left up to whoever it is that's doing their work, right, because like if you're copying over a cell network you need to do something very different as if you're broadcasting over a terrestrial XM or something like that, you need to do something very different for satellite. >> If you're at the edge on a device, would you be actually running, like you were saying earlier, structured streaming, with the prediction? >> Right, I don't think you would use structured streaming per se on the edge device, but essentially there would be a lot of code share between structured streaming and the code that you'd be using on the edge device. And it's being vectored out now so that we can have this code sharing and Spark machine learning. And you would use structured streaming maybe on the training side, and then on the serving side you would use your custom local code. >> Okay, so tell us a little more about Spark ML today and how we can democratize machine learning, you know, for a bigger audience. >> Right, I think machine learning is great, but right now you really need a strong statistical background to really be able to apply it effectively. And we probably can't get rid of that for all problems, but I think for a lot of problems, doing things like hyperparameter tuning can actually give really powerful tools to just like regular engineering folks who, they're smart, but maybe they don't have a strong machine learning background. And Spark's ML pipelines make it really easy to sort of construct multiple stages, and then just be like, okay, I don't know what these parameters should be, I want you to do a search over what these different parameters could be for me, and it makes it really easy to do this as just a regular engineer with less of an ML background. >> Would that be like, just for those of us who are, who don't know what hyperparameter tuning is, that would be the knobs, the variables? >> Yeah, it's going to spin the knobs on like our regularization parameter on like our regression, and it can also spin some knobs on maybe the engram sizes that we're using on the inputs to something else, right. And it can compare how these knobs sort of interact with each other, because often you can tune one knob but you actually have six different knobs that you want to tune and you don't know, if you just explore each one individually, you're not going to find the best setting for them working together. >> So this would make it easier for, as you're saying, someone who's not a data scientist to set up a pipeline that lets you predict. >> I think so, very much. I think it does a lot of the, brings a lot of the benefits from sort of the SciPy world to the big data world. And SciPy is really wonderful about making machine learning really accessible, but it's just not ready for big data, and I think this does a good job of bringing these same concepts, if not the code, but the same concepts, to big data. >> The SciPy, if I understand, is it a notebook that would run essentially on one machine? >> SciPy can be put in a notebook environment, and generally it would run on, yeah, a single machine. >> And so to make that sit on Spark means that you could then run it on a cluster-- >> So this isn't actually taking SciPy and distributing it, this is just like stealing the good concepts from SciPy and making them available for big data people. Because SciPy's done a really good job of making a very intuitive machine learning interface. >> So just to put a fine sort of qualifier on one thing, if you're doing the internet of things and you have Spark at the edge and you're running the model there, it's the programming model, so structured streaming is one way of programming Spark, but if you don't have structured streaming at the edge, would you just be using the core batch Spark programming model? >> So at the edge you'd just be using, you wouldn't even be using batch, right, because you're trying to predict individual events, right, so you'd just be calling predict with every new event that you're getting in. And you might have a q mechanism of some type. But essentially if we had this batch, we would be adding additional latency, and I think at the edge we really, the reason we're moving the models to the edge is to avoid the latency. >> So just to be clear then, is the programming model, so it wouldn't be structured streaming, and we're taking out all the overhead that forced us to use batch with Spark. So the reason I'm trying to clarify is a lot of people had this question for a long time, which is are we going to have a different programming model at the edge from what we have at the center? >> Yeah, that's a great question. And I don't think the answer is finished yet, but I think the work is being done to try and make it look the same. Of course, you know, trying to make it look the same, this is Boosh, it's not like actually barking at us right now, even though she looks like a dog, she is, there will always be things which are a little bit different from the edge to your cluster, but I think Spark has done a really good job of making things look very similar on single node cases to multi node cases, and I think we can probably bring the same things to ML. >> Okay, so it's almost time, we're coming back, Spark took us from single machine to cluster, and now we have to essentially bring it back for an edge device that's really light weight. >> Yeah, I think at the end of the day, just from a latency point of view, that's what we have to do for serving. For some models, not for everyone. Like if you're building a website with a recommendation system, you don't need to serve that model like on the edge node, that's fine, but like if you've got a car device we can't depend on cell latency, right, you have to serve that in car. >> So what are some of the things, some of the other things that IBM is contributing to the ecosystem that you see having a big impact over the next couple years? >> So there's a lot of really exciting things coming out of IBM. And I'm obviously pretty biased. I spend a lot of time focused on Python support in Spark, and one of the most exciting things is coming from my co-worker Brian, I'm not going to say his last name in case I get it wrong, but Brian is amazing, and he's been working on integrating Arrow with Spark, and this can make it so that it's going to be a lot easier to sort of interoperate between JVM languages and Python and R, so I'm really optimistic about the sort of Python and R interfaces improving a lot in Spark and getting a lot faster as well. And we're also, in addition to the Arrow work, we've got some work around making it a lot easier for people in R and Python to get started. The R stuff is mostly actually the Microsoft people, thanks Felix, you're awesome. I don't actually know which camera I should have done that to but that's okay. >> I think you got it! >> But Felix is amazing, and the other people working on R are too. But I think we've both been pursuing sort of making it so that people who are in the R or Python spaces can just use like Pit Install, Conda Install, or whatever tool it is they're used to working with, to just bring Spark into their machine really easily, just like they would sort of any other software package that they're using. Because right now, for someone getting started in Spark, if you're in the Java space it's pretty easy, but if you're in R or Python you have to do sort of a lot of weird setup work, and it's worth it, but like if we can get rid of that friction, I think we can get a lot more people in these communities using Spark. >> Let me see, just as a scenario, the R server is getting fairly well integrated into Sequel server, so would it be, would you be able to use R as the language with a Spark execution engine to somehow integrate it into Sequel server as an execution engine for doing the machine learning and predicting? >> You definitely, well I shouldn't say definitely, you probably could do that. I don't necessarily know if that's a good idea, but that's the kind of stuff that this would enable, right, it'll make it so that people that are making tools in R or Python can just use Spark as another library, right, and it doesn't have to be this really special setup. It can just be this library and they point out the cluster and they can do whatever work it wants to do. That being said, the Sequel server R integration, if you find yourself using that to do like distributed computing, you should probably take a step back and like rethink what you're doing. >> George: Because it's not really scale out. >> It's not really set up for that. And you might be better off doing this with like, connecting your Spark cluster to your Sequel server instance using like JDBC or a special driver and doing it that way, but you definitely could do it in another inverted sort of way. >> So last question from me, if you look out a couple years, how will we make machine learning accessible to a bigger and bigger audience? And I know you touched on the tuning of the knobs, hyperparameter tuning, what will it look like ultimately? >> I think ML pipelines are probably what things are going to end up looking like. But I think the other part that we'll sort of see is we'll see a lot more examples of how to work with certain kinds of data, because right now, like, I know what I need to do when I'm ingesting some textural data, but I know that because I spent like a week trying to figure out what the hell I was doing once, right. And I didn't bother to write it down. And it looks like no one else bothered to write it down. So really I think we'll see a lot of tools that look very similar to the tools we have today, they'll have more options and they'll be a bit easier to use, but I think the main thing that we're really lacking right now is good documentation and sort of good books and just good resources for people to figure out how to use these tools. Now of course, I mean, I'm biased, because I work on these tools, so I'm like, yeah, they're pretty great. So there might be other people who are like, Holden, no, you're wrong, we need to rethink everything. But I think this is, we can go very far with the pipeline concept. >> And then that's good, right? The democratization of these things opens it up to more people, you get more creative people solving more different problems, that makes the whole thing go. >> You can like install Spark easily, you can, you know, set up an ML pipeline, you can train your model, you can start doing predictions, you can, people that haven't been able to do machine learning at scale can get started super easily, and build a recommendation system for their small little online shop and be like, hey, you bought this, you might also want to buy Boosh, he's really cute, but you can't have this one. No no no, not this one. >> Such a tease! >> Holden: I'm sorry, I'm sorry. >> Well Holden, that will, we'll say goodbye for now, I'm sure we will see you in June in San Francisco at the Spark Summit, and look forward to the update. >> Holden: I look forward to chatting with you then. >> Absolutely, and break a leg this afternoon at your presentation. >> Holden: Thank you. >> She's Holden Karau, I'm Jeff Frick, he's George Gilbert, you're watching The Cube, we're at Big Data SV, thanks for watching. (upbeat music)