>> Announcer: Live from San Diego, California, it's theCUBE, covering KubeCon + CloudNativeCon. Brought to you by Red Hat, the Cloud Native Computing Foundation and its ecosystem partners. >> Welcome to the third day of wall-to-wall coverage here at Kubecon + CloudNativeCon 2019 in San Diego. I am your host for the three days of coverage, Stu Miniman. Joining me this morning is Justin Warren. And happy to welcome back to the program, Erin Boyd who's a senior principal software engineer at Red Hat. Erin, thanks so much for joining us. >> Thanks for having me. >> All right, so we had a chance to catch up in Barcelona on theCUBE there. Storage is definitely one of the faster moving areas of this ecosystem over the last two years. Why don't we start with, really, the event? So, you know, as I said, we're in day three but day zero there were a whole lot of things we had. Some of your peers at Red Hat have talked about OpenShift Commons, but storage, to my understanding had a couple of things going on. Why don't you share with our audience a little bit of that? >> Sure, so we had a SIG face-to-face for Kubernetes, it was probably one of the best attended. We had to cap the number of attendees, so about 60 different people came to talk about the future of Kubernetes in storage, and what we need to be doing to meet our customers' needs. In conjunction with that, there was a parallel session called CNS Days, which is Container Native Storage Days. That event is very customer focused, so I really enjoyed bouncing between the two of them. To go from the hypothetical, programming, architecture view, straight to what customers in the enterprise are looking at and doing, and what their real needs are. >> So from that SIG, can you actually share a little bit of where we are, where some of the requests are? We know storage is never one way to fix it, there's been some debates, there's a couple different ways to do... I mean, traditional storage, you've got block, file, and object. Cloud storage, there are more options in cloud storage today than there was, if I was to configure a server, or buy a storage array in my own data center. So where are we, what are those asks? What's on the roadmap there? >> Right, so I think for the past five years, we've been really focused on being mindful of what APIs are common across all the vendors. I think we want to ensure that we're not excluding any vendors from being part of this ecosystem. And so, with that, we've created the basis of things like persistent volumes, persistent volumes claims, storage classes to automate that, storage quotas to be able to have management and control over it. So I think now we're looking to the next evolution of... As the model's maturing, and people are actually running stateful applications on Kubernetes, we need to be addressing their needs. So things like snapshotting, eventually volume cloning, which has just gone in, and migrating. All these type of things that exist within the data plane are going to be the next evolution of things we look at in the SIG. >> Yeah, so one criticism that's been mentioned about Kubernetes a few times, that one, it's a bit complicated. But also, it didn't really deal that well with stateful sets. Stateful data management has always been, it's been a little bit lacking. That seems to have pretty much been sorted out now. As you mentioned, there's a lot more work being done on storage operators. But you're talking about some of these data management features that operators from other paradigms are kind of used to being there. When you're thinking about moving workloads to Kubernetes, or putting in new workloads on Kubernetes, if you're unsure about, "Well, will I be able to operate this in the same way that I did things before?" How do you think people should be thinking about those kind of data services in Kubernetes? >> So I think it's great that you mentioned operators. Because that was one of the key things when Rook came into the landscape, to be able to lower the complexity of taking something that requires physical storage and compute, geography, node selection. All those things, it helped people who were used to just the cloud model. I create a PVC, it's a request for storage, Amazon magically fulfills it. I don't know what's backing it. To be able to take these more complex storage systems and deploy them within the ecosystem, it also does a good job supporting our Brownfield customers, because not every customer that's coming to Kubernetes is green. So it's important that we understand that some customers want to keep their data on-prem, maybe burst to the cloud to leverage those services, but then keep their data close to home. So operators help facilitate that. >> Yeah, Erin, I hesitate a little bit to ask this, but I'm wondering if you can do a little compare, contrast for us, for what the industry had done back in OpenStack days? When I looked at storage, every traditional storage company certified their environment for OpenStack. On a storage standpoint, it feels like a different story to me when I hear about the ecosystem of operators in OpenStack. So I know you know this space, so maybe you can give us a little bit of what we learned in the past. What's similar, what's different? >> Right, well I think one of the benefits is we have a lot of the same key players. As you may know, OpenShift has pivoted from Gluster to Ceph, Ceph being the major backer of OpenStack. So we're able to take some of that technical debt, and learn our lessons from things we could improve, and apply those things within Kubernetes. I just think that it's a little slower migration, because in OpenStack, like you said, we had certification, there were different drivers. And we're trying to learn from, maybe, I wouldn't even call those mistakes, but, how can we better automate this? What can we do from an operational perspective to make it easier? >> Well I think because one of the... It felt like we were kind of taking some older models and... I'm testing it, I'm adding it. The ecosystem for operators here is different. Many of these, we're talking very much software-driven solutions. It's built for container architectures, so it's understandable that it might take a little bit longer because it's a different paradigm. >> Right, well, and I think the certification kind of... It wasn't an inhibitor but it certainly took a lot of time. And I think our take was on... We used to have all the storage providers be entry providers within Kubernetes. And with CSI, we have since started to redo the plugins and the sidecars, and move that out of core. So then the certification kind of falls outside of that instead of being more tightly wound into the platform. And I think it will allow us to have a lot more flexibility. Instead of waiting on each release, vendors can create operators, certify them themselves, have them in their own CSI driver, and move at the pace that they need to move. >> So how do you balance that need for Kubernetes to be a common operating platform that people can build on with each vendor's desire to provide their own unique capabilities that they think that they do particularly well? That's why they charge the money that they do, because they think that theirs is the best storage ever. How do you balance that tension between the need for a standard platform and to make it interoperable, but still allowing the flexibility for people to have their own kind of innovation in there? >> So when we created the storage class, for instance, to be able to create a service level over storage, to be able to provide the provisioner that we're going to use, we made the specification of that section completely opaque. And what that allowed us to do is that when vendors wrote their provisioners and now their CSI drivers, allowed them to feed in different attributes of the storage that they want to leverage, that don't necessarily have to be in core Kubernetes. So it provided a huge amount of flexibility on that. The other side of that, though, is, the feeback we get from real users is "I need backup and recovery, and I need DR, and I need that across the platform." So I really think as we look to scale this out, we have to be looking at the commonalities between all storage and bringing those APIs into Kubernetes. >> One of the things I've really liked to see in this ecosystem over the last year or so, and really highlighted at this show, we're talking a lot more about workloads and applications and how those... What works today and where we're growing. Can you speak a little bit from your world as to where we are, what's working great, what customers are deploying, and a little bit, the road map of where we still need to go? >> Sure, I think workloads are key. I mean, I think that we have to focus on the actual end-to-end delivery of that, and so we have to figure out a way that we can make the data more agile, and create interfaces to really enable that, because it's very unlikely that an enterprise company is going to rely on one cloud or stay with one cloud, or want their data in one cloud. They're going to want to have the flexibility to leverage that. So as we enable those workloads, some are very complex. We started with, "Hey, I just want to containerize my application and get it running. Now I want to have some sort of state, which is persistent storage, and now I want to be able to scale that out across n number of clusters." That's where the workloads become really important. And long term, where we need policy to automate that. My pod goes down, I restart it, it needs to know that because of, maybe, the data that that workload's producing, it can only stay in this geographical region. >> Yeah, we talk about multicloud. You mentioned data protection, data protection is something I need to do across the board. Security is something I need to do across the board. My automation needs to take all that into account. How's Red Hat helping customers get their arms around that challenge? >> Yeah, so I think Red Hat really does take a holistic view in making sure that we provide a very consistent, secure platform. I think that's one of the things that you see when you come on to OpenShift, for instance, or OKR, that you're seeing security tightened a little bit more, to ensure that you're running in the best possible way that you can, to protect your data. And then, the use of Rook Ceph, for instance, Ceph provides that universal backplane, where if you're going to have encryption or anything like that, you know it's going to be the same across that. >> It sounds like there's an opportunity here for people new to Kubernetes who have been doing things in a previous way. There's a little bit of reticence from this community to understand enterprise, they're like, "Well, actually, you're kind of doing it wrong. It's slow and inflexible." There's actually a lot of lessons that we've learned in enterprise, particularly around these workloads. Having security, having backup in DR. In the keynote this morning, there was a lot of discussion about the security that either is in Kubernetes, and some parts it's kind of lacking. I think there's a lot that both of these communities can learn from each other, so I'm seeing a lot of moves of late to be a little bit more welcoming to some people who are coming to Kubernetes from other ecosystems. To be able to bring the ideas that they have that... We've already learned these lessons before, we can take some of that knowledge and bring it into Kubernetes to help us to do that better. Do you see Red Hat bringing a lot of that expereience in its work... Red Hat's been around for quite some time now, so you've done a lot of this already. Are you bringing all of that knowledge into Kubernetes and sharing it with the ecosystem? >> Absolutley, and just like Stu pointed out, I mean, OpenStack was a big part of our evolution, and security within RHEL, and I think we absolutely should take those lessons learned and look to how we do protect our customers' data, and make sure that the platform, Kubernetes itself and as we evolve OpenShift, can provide that, and ways that we can certify that. >> Erin, you're meeting with a lot of customers. You were talking about the Day Zero thing. What's top of mind for your customers? We talk about, that Kubernetes has crossed the chasm but to get the vast majority, there's still lots of work to do. We need to, as an industry, make things simpler. What's working well, and what are some of the challenges from the customers that you've talked to? >> So I think, if you walk in, across the hall, and you see how many vendors are there, it's trying to get a handle on what I should even be doing. And as the co-lead of the CNCF Storage SIG, I think that's one of the initiatives that we take very seriously. So in addition to a storage whitepaper, we've been working on use cases that define, when should I use a data store? When should I use object? Why would I want to use file? And then really taking these real-world examples, creating use cases and actual implementations so someone can, "Oh, that's similar to my workload." Here are some tools to accelerate understanding how to get that set up. And also creating those guard rails from an architectural standpoint. You don't want to go down this path, that's not right for your workload. So we're hoping to at least provide an education around containerized storage that'll help customers. >> Yeah, I'm just curious. I think back ten years ago, I was working for a large storage company. We were having some of these same conversations. So is it very different now in the containerized, multicloud world? Or are some of the basic decision tree discussions around block, file, and object and application the same as we might have been having a decade ago? >> I think we're starting to just touch on those, and I'm glad that you brought up object. That was one of the things I talked about in Barcelona, and we actually talked about at the face-to-face. To me, it's kind of the missing piece of storage today in Kubernetes, and I think we're finally starting to see that more customers are asking for that and realizing that's an important workload to be able to support at its core. So I think, yes, we're having the same conversations again, but certainly in a different context. >> Yeah, I mean, back in the day, it was, the future is object but we don't know how we'd get there. If you look behind the scenes in most public clouds, object's running a lot of what's there. All right, Erin, I want to give you the final word. KubeCon 2019, from that storage perspective. What should people watching take away? >> That we're only beginning with storage, yeah. We still have a lot of work to do, but I think it's a wonderful community and vibrant, and I think there'll be a lot of changes in the coming years. >> All right. Well, definitely a vibrant ecosystem. Erin, thank you so much for all the updates. We'll be back with more coverage here, for Justin Warren. I'm Stu Miniman. Thank you for watching theCUBE. (techno music)

>> Live, from Barcelona, Spain, it's the theCUBE, covering KUBECON and CloudNativeCon Europe 2019. Brought to you by RedHat, the Cloud Native Computing Foundation, and the Ecosystem Partners. >> Welcome back to theCUBE. I'm Stu Miniman. My co-host, Corey Quinn. 7700 here in Barcelona, Spain, for KUBECON, CLOUDNATIVECON. Happy to welcome to the program a first-time guest, Erin Boyd, who is a senior Principal Software Engineer in the office of the CEO of RedHat. Erin, thanks so much for joining us. >> Yeah, thanks for having me. >> Alright, so just a couple of weeks ago, I know I was in Boston, you probably were too, >> Yep. >> For RedHat Summit. Digging into a lot of the pieces. You focus on multi-cloud and storage. Tell us a little bit about, you know, your role, and what you're doing here at the KUBECON show. >> Sure, I'd be happy to. So for over a year now, RedHat's really been kind of leading the pack on hybrid cloud. You know, allowing customers to have more choice, you know, with both public and private cloud offerings. And, of course, OpenShift being our platform built on Kubernetes, we believe that should be the consistent API in which we have Federation. Yeah, so Erin, I got to talk to quite a few OpenShift customers at RedHat Summit. It was really how they're using that as a lever to help them really gain agility in their application deployment. But, let's start for a second, without getting too fanatic, you say hybrid cloud. What does that mean to your customers? You know, RedHat has a long legacy of, well, lives everywhere. So, public cloud, private cloud, hosting provider, all of the environments, you, RedHat, Enterprise, Linux, can live there. So in your space, what does hybrid cloud mean? >> So, hybrid cloud, I think follows a model of real. It's everywhere. So it's having OpenShift run on top of that and being able to have the application portability that you would expect. Along with the application portability, which is my focus, is having the data agility within those applications. >> Alright, how do you wind up approaching a situation where an app is now agile enough to move between providers almost seamlessly, without having it, I guess, descend down to the lowest common denominator that all providers that it's on are going to provide? I mean, at some point, doesn't that turn into treating the cloud as a place to just run your either instances or containers, and not taking advantage of, I guess, the platform level services? >> Sure, so I think that the API should expose those choices, I don't think it's a one size fits all when we talk about, you know, if you move your application maybe your data doesn't necessarily have to move. So part of the core functionality the Federation is meant to provide, which has been renamed Kubefed since Summit, is that you have the choice within that. And, you know, defining policies around the way we do this. So, perhaps your application is agile enough to span three different clouds, but due to data privacy, you want to keep your data on prem. So, Kubefed should enable you to have that choice. >> You know, so you know, help us dig down a little bit in the storage, you know, environment here, you know. >> Sure. I go back and I worked for a very large storage company that was independent before it got bought for a very large sum of money. But, we had block and file storage. And mostly, that you know, lived in a box, or in a certain application. >> Right. You know, the future, we always talked that there's going to be this wonderful object storage and actually it's designed to be, you know, we'll shard it, we'll spread it around >> Right. And it can live in lots of places. Cloud, a lot of times has that underneath it, so you know, have we started to you know, cross that gap of you know, that mythical nirvana of where say, you know, storage should actually live up to that distributive architecture that we're all looking for. >> Right, so with Kubernetes, the history is, we started off with only file systems. Block is something very new within the last couple releases that I actually personally worked on. The next piece that we're doing at Red Hat is leading the charge to create CRDs for object storage. So it's defining those APIs so customers can dynamically provision and manage their object storage with that. In addition, we recently acquired a company called NooBaa that does exactly that. They're able to have that data mobility through object buckets across many clouds doing the sharding and replication with the ability to dedupe. And that's super important because it opens up for our customers to have image streams, photos, things like that that they typically use within an enterprise, and quickly move the data and copy it as they need to. >> Yeah, so I've actually talked to the Noovaa team. I would joke with them that, didn't they deduplicate, couldn't they deduplicate their name 'cause it's like Noovaa. >> (laughs) yeah. >> So you know, plenty of vowels there. But, right, storage built for the cloud world is, you know, what we're talking about there. >> Right. >> How's that different from some of the previous storage solutions that we've been dealing with? >> So I think before, we were trying to maybe make fit what didn't work. That's not to say that file and block aren't important. I mean, having local storage for a high performance application is absolutely critical. So I think we're meeting the market where it is. It's dependent on the behavior of the application. and we should be able to provide that. And applications that primarily run in the cloud and need that flexibility, we should be offering object as a first-class citizen, and that's why our work with those CRDs is really critical. >> What is the customer need that drives this? Historically, with my own work with object stores, I tend to view that as almost exclusively accessed via HTTP end points. And at that point, it almost doesn't matter where that lives, as long as the networking and security and latency requirements are being met. What is it that's driving this as making it a first-class citizen built in to Kubernetes itself, the Rook? >> So it allows us to create the personas that we need. So it allows an administrator to administrate storage, just like they would normally with your persistent volume, persistent volume claims and quotas. And then it abstracts the details of, for instance, including that URL in your application. We use a config map within the app so the user doesn't have access necessarily to your keys in the cloud. It also creates a user so you're able to manage users like you would normal objects, which is a little bit different than the PV PVC, and that's why we feel like you know, it's important to have a CRD that defines object in that sense because it is a little bit different. >> All right, so Erin, is this Rook we're talking about then, is, you know, Rook, did I understand, I think got to 1.0, just got released. >> Yeah. >> You know, give us the update on what Rook is, you know, how that fits with this conversation we've been having. >> Right. You know, where we are with the maturity of it. And Rook, as was on the keynote this morning, you know, is a great CNCF project with a really healthy community behind it. One of the provisioners we've created as part of those object CRDs is a Rook provisioner for CEF block, or excuse me, CEF object. We also have an s3 provisioner. So, you know, we hope to have, just like we had external provisioners in Kubernetes, use, you know, allow for the same contribution from the community for those. >> Okay, yeah, there, I remember a couple of years ago at the show, this fixing storage for containers in Kubernetes was something that was a little bit contention in there, and there were a few different projects out there. >> Right. >> For that, you know, where are we with that? We understand that it's never, you know, one solution for every single use case. You know, you already talked about, you know, block file and object. >> Right. >> And how there's going to be a you know, a spectrum of options. >> Sure and so I think there's lots of things to fix. >> Yeah. >> When you talk about that. One of the key things that Rook offered was the ability to ease the deployment of the storage and administration of it, and, as you know, Rook you know, has a plethora of different storage systems that it provides. And, you know, what we're really pushing at RedHat, which I think is important, is having, you know, operators. Like the operator hub that was released with OpenShift 4.0. Rook will be an operator in there. So what that allows is for more automation and true scaling. 'Cause that's where we want to get to with hybrid cloud. If you're managing 10,000 clusters, you cannot do that manually. So having Rook, having operators, and automating the storage piece underneath is really critical to make it now-scale happen. >> Forgive my ignorance. When you say that Rook winds up exposing, for example, now an object store underneath. Is that it's own pile of disks on a system somewhere that it's running? Is it wrapping around object store provided by other cloud providers? Is it something else entirely? What is the, where do the actual drives that hold my data, when I'm using Rook's object store, live? So with Rook today, the object storage that it uses is CEF object. So it exposes the ability to create, you know the CEF components underneath, which Rook can lay down and then expose the object piece of that. So that's the first provisioner in there, yep. >> Wonderful. >> Alright, so I guess when I think about object storage, for years it's been, well, I've got s3 compatibility. And that's kind of the big thing. >> Yep. Is Rook s3 compatible then? Is it, you know, giving more flexibility to users to make this the standard in a cloud native environment? Help us, you know, put a fine as to what this is and isn't. >> Yeah, that's a great question, actually, and we get asked it often. So one of the first provisioners we did is just a proof the concept was an s3, a generic s3 provisioner. And of course, CEF is s3 compliant, so it also does that, but you know, there isn't a standard for object. So most providers of object are s3 compatible. We found it very easy to take off the s3 provisioner we created to create the CEF one. There wasn't much differentiation, which means it's a great pattern for anyone to want to onboard. >> Yeah. Do you find that as s3 itself, and of course, it's competitors of other cloud providers, become more capable, you're starting to see differentiation. Now easy example would be with some of the object storage tiers, where there's increased latency on retrievals. In some cases, as little as five minutes, or as much as 12 hours. Other providers, like Google Cloud, for example, or Azure, have consistent retrieval times on their archive storage. As an easy example, is that something that you're going to start seeing divergence on as object storage becomes smarter by, I guess, all of the providers as they race each other to improve their products. >> Absolutely. I think tiering is one of the facets of object that's really critical. And you know, of course, as we spoke earlier, it's physics, you know, and having data consistency at that very low threshold is important. So, you know, using the storage for what it's worth. Using the best tools, and pulling object into the ecosystem is part of that. >> Yeah, Erin, is there anything that differentiates kind of Kubernetes storage from, you know, what people are familiar with in the past? >> I think Kubernetes storage continues to evolve. The more we learn about how people use Kubernetes, and their needs, I think we listen closely to the community and we develop against that. >> Okay, I guess the other thing is, you know, what kind of feedback are you getting from customers? Where are we along this maturation journey. You know, my history is you know, I worked when we had to fix networking and storage in virtualized environment, and it took about a decade. We're five years into Kubernetes. It feels like we've, you know, accelerated that based on what we've done in the past, but you know, definitely, you know, when it first started, it was you know, let's put stateless stuff in containers and you know, storage will be an afterthought. >> Right. >> Or something that was kind of a side car over here where you had your repository. >> Right. And I think that's the beauty of Kubefed, is that in order to have true hybrid cloud, and have Federation, we have to come together in consensus with both network compute and storage. So it really brings the story full circle. >> Perfect. What do you think right now customers are having their biggest challenges with, as they start wrapping their minds around this new way of thinking? I mean, again, it's easy for a tiny start-up, it's Twitter for Pets, or something like that, to spin off in a pure cloud native way, but larger companies with this legacy concept known as a business model that might involve turning a profit, generally predate cloud, and have done an awful lot of stuff on the data center. What are they seeing as currently being limiting factors on their digital transformation? >> So with Kubernetes just being five years old, as we celebrate the birthday today, I think customers are also maturing. You know, they're entering the landscape, learning about Kubernetes, learning how to containerize, you know, lift and ship their applications, and then they're running up, to costs, right? And lock-ins and things they want to avoid. And that's really where we in the community want to provide a platform and a runway for them to have that choice. >> Alright. Erin, any customer successes that you can share with us, either about the operator or about work specifically? >> Certainly not with Federation. We haven't released it. It will come out in OpenShift 4.2, so we don't have any customer success stories yet, but I would say definitely it's a request, and you know, we're asking customers about it, and if they're interested. And you will find many times maybe they're not familiar with the word Federation, but they're definitely interested in that use case. >> Okay, how's the general feel. You know, what kind of feedback are you getting from customers so far, things that you're excited about that are happening here at the show? >> I'm just excited that Kubernetes is kind of growing up. And it's you know, becoming a true enterprise-level project that customers rely on, and build their business on. >> Well, Erin Boyd, really appreciate you joining us, sharing all the updates. Look forward to the upcoming release, and definitely get to follow up with you soon, to hear about those customers as they start rolling it out. >> Alright, great. Thank you. >> Alright. For Corey Quinn, I'm Stu Miniman, here at KUBECON, CLOUDNATIVECON 2019, Barcelona, Spain. Thanks for watching theCUBE (techno music)