(upbeat music) >> Announcer: theCUBE presents KubeCon and CloudNativeCon Europe 2022, brought to you by Red Hat, the Cloud Native Computing Foundation, and its ecosystem partners. >> Welcome to Valencia, Spain, in KubeCon CloudNativeCon Europe 2022. I'm your host Keith Townsend, along with Paul Gillon. And we're going to talk to some amazing folks. But first Paul, do you remember your college days? >> Vaguely. (Keith laughing) A lot of them are lost. >> I think a lot of mine are lost as well. Well, not really, I got my degree as an adult, so they're not that far past. I can remember 'cause I have the student debt to prove it. (both laughing) Along with us today is Kenneth Hoste, systems administrator at Ghent University, and Marcel Hild, senior manager software engineering at Red Hat. You're working in office of the CTO? >> That's absolutely correct, yes >> So first off, I'm going to start off with you Kenneth. Tell us a little bit about the research that the university does. Like what's the end result? >> Oh, wow, that's a good question. So the research we do at university and again, is very broad. We have bioinformaticians, physicists, people looking at financial data, all kinds of stuff. And the end result can be very varied as well. Very often it's research papers, or spinoffs from the university. Yeah, depending on the domain I would say, it depends a lot on. >> So that sounds like the perfect environment for cloud native. Like the infrastructure that's completely flexible, that researchers can come and have a standard way of interacting, each team just use it's resources as they would, the Navana for cloud native. >> Yeah. >> But somehow, I'm going to guess HPC isn't quite there yet. >> Yeah, not really, no. So, HPC is a bit, let's say slow into adopting new technologies. And we're definitely seeing some impact from cloud, especially things like containers and Kubernetes, or we're starting to hear these things in HPC community as well. But I haven't seen a lot of HPC clusters who are really fully cloud native. Not yet at least. Maybe this is coming. And if I'm walking around here at KubeCon, I can definitely, I'm being convinced that it's coming. So whether we like it or not we're probably going to have to start worrying about stuff like this. But we're still, let's say, the most prominent technologies of things like NPI, which has been there for 20, 30 years. The Fortran programming language is still the main language, if you're looking at compute time being spent on supercomputers, over 1/2 of the time spent is in Fortran code essentially. >> Keith: Wow. >> So either the application itself where the simulations are being done is implemented in Fortran, or the libraries that we are talking to from Python for example, for doing heavy duty computations, that backend library is implemented in Fortran. So if you take all of that into account, easily over 1/2 of the time is spent in Fortran code. >> So is this because the libraries don't migrate easily to, distributed to that environment? >> Well, it's multiple things. So first of all, Fortran is very well suited for implementing these type of things. >> Paul: Right. >> We haven't really seen a better alternative maybe. And also it'll be a huge effort to re-implement that same functionality in a newer language. So, the use case has to be very convincing, there has to be a very good reason why you would move away from Fortran. And, at least the HPC community hasn't seen that reason yet. >> So in theory, and right now we're talking about the theory and then what it takes to get to the future. In theory, I can take that Fortran code put it in a compiler that runs in a container? >> Yeah, of course, yeah. >> Why isn't it that simple? >> I guess because traditionally HPC is very slow at adopting new stuff. So, I'm not saying there isn't a reason that we should start looking at these things. Flexibility is a very important one. For a lot of researchers, their compute needs are very picky. So they're doing research, they have an idea, they want you to run lots of simulations, get the results, but then they're silent for a long time writing the paper, or thinking about how to, what they can learn from the results. So there's lots of peaks, and that's a very good fit for a cloud environment. I guess at the scale of university you have enough diversity end users that all those peaks never fall at the same time. So if you have your big own infrastructure you can still fill it up quite easily and keep your users happy. But this busty thing, I guess we're seeing that more and more or so. >> So Marcel, talk to us about, Red Hat needing to service these types of end users. That it can be on both ends I'd imagine that you have some people still in writing in Fortran, you have some people that's asking you for objects based storage. Where's Fortran, I'm sorry, not Fortran, but where is Red Hat in providing the underlay and the capabilities for the HPC and AI community? >> Yeah. So, I think if you look at the user base that we're looking at, it's on this spectrum from development to production. So putting AI workloads into production, it's an interesting challenge but it's easier to solve, and it has been solved to some extent, than the development cycle. So what we're looking at in Kenneth's domain it's more like the end user, the data scientist, developing code, and doing these experiments. Putting them into production is that's where containers live and thrive. You can containerize your model, you containerize your workload, you deploy it into your OpenShift Kubernetes cluster, done, you monitor it, done. So the software developments and the SRE, the ops part, done, but how do I get the data scientist into this cloud native age where he's not developing on his laptop or on a machine, where he SSH into and then does some stuff there. And then some system admin comes and needs to tweak it because it's running out of memory or whatnot. But how do we take him and make him, well, and provide him an environment that is good enough to work in, in the browser, and then with IDE, where the workload of doing the computation and the experimentation is repeatable, so that the environment is always the same, it's reliable, so it's always up and running. It doesn't consume resources, although it's up and running. Where it's, where the supply chain and the configuration of... And the, well, the modules that are brought into the system are also reliable. So all these problems that we solved in the traditional software development world, now have to transition into the data science and HPC world, where the problems are similar, but yeah, it's different sets. It's more or less, also a huge educational problem and transitioning the tools over into that is something... >> Well, is this mostly a technical issue or is this a cultural issue? I mean, are HPC workloads that different from more conventional OLTP workloads that they would not adapt well to a distributed containerized environment? >> I think it's both. So, on one hand it's the cultural issue because you have two different communities, everybody is reinventing the wheel, everybody is some sort of siloed. So they think, okay, what we've done for 30 years now we, there's no need to change it. And they, so it's, that's what thrives and here at KubeCon where you have different communities coming together, okay, this is how you solved the problem, maybe this applies also to our problem. But it's also the, well, the tooling, which is bound to a machine, which is bound to an HPC computer, which is architecturally different than a distributed environment where you would treat your containers as kettle, and as something that you can replace, right? And the HPC community usually builds up huge machines, and these are like the gray machines. So it's also technical bit of moving it to this age. >> So the massively parallel nature of HPC workloads you're saying Kubernetes has not yet been adapted to that? >> Well, I think that parallelism works great. It's just a matter of moving that out from an HPC computer into the scale out factor of a Kubernetes cloud that elastically scales out. Whereas the traditional HPC computer, I think, and Kenneth can correct me here is, more like, I have this massive computer with 1 million cores or whatnot, and now use it. And I can use my time slice, and book my time slice there. Whereas this a Kubernetes example the concept is more like, I have 1000 cores and I declare something into it and scale it up and down based on the needs. >> So, Kenneth, this is where you talked about the culture part of the changes that need to be happening. And quite frankly, the computer is a tool, it's a tool to get to the answer. And if that tool is working, if I have a 1000 cores on a single HPC thing, and you're telling me, well, I can't get to a system with 2000 cores. And if you containerized your process and move it over then maybe I'll get to the answer 50% faster maybe I'm not that... Someone has to make that decision. How important is it to get people involved in these types of communities from a researcher? 'Cause research is very tight-knit community to have these conversations and help that see move happen. >> I think it's very important to that community should, let's say, the cloud community, HPC research community, they should be talking a lot more, there should be way more cross pollination than there is today. I'm actually, I'm happy that I've seen HPC mentioned at booths and talks quite often here at KubeCon, I wasn't really expecting that. And I'm not sure, it's my first KubeCon, so I don't know, but I think that's kind of new, it's pretty recent. If you're going to the HPC community conferences there containers have been there for a couple of years now, something like Kubernetes is still a bit new. But just this morning there was a keynote by a guy from CERN, who was explaining, they're basically slowly moving towards Kubernetes even for their HPC clusters as well. And he's seeing that as the future because all the flexibility it gives you and you can basically hide all that from the end user, from the researcher. They don't really have to know that they're running on top of Kubernetes. They shouldn't care. Like you said, to them it's just a tool, and they care about if the tool works, they can get their answers and that's what they want to do. How that's actually being done in the background they don't really care. >> So talk to me about the AI side of the equation, because when I talk to people doing AI, they're on the other end of the spectrum. What are some of the benefits they're seeing from containerization? >> I think it's the reproducibility of experiments. So, and data scientists are, they're data scientists and they do research. So they care about their experiment. And maybe they also care about putting the model into production. But, I think from a geeky perspective they are more interested in finding the next model, finding the next solution. So they do an experiment, and they're done with it, and then maybe it's going to production. So how do I repeat that experiment in a year from now, so that I can build on top of it? And a container I think is the best solution to wrap something with its dependency, like freeze it, maybe even with the data, store it away, and then come to it back later and redo the experiment or share the experiment with some of my fellow researchers, so that they don't have to go through the process of setting up an equivalent environment on their machines, be it their laptop, via their cloud environment. So you go to the internet, download something doesn't work, container works. >> Well, you said something that really intrigues me you know in concept, I can have a, let's say a one terabyte data set, have a experiment associated with that. Take a snapshot of that somehow, I don't know how, take a snapshot of that and then share it with the rest of the community and then continue my work. >> Marcel: Yeah. >> And then we can stop back and compare notes. Where are we at in a maturity scale? Like, what are some of the pitfalls or challenges customers should be looking out for? >> I think you actually said it right there, how do I snapshot a terabyte of data? It's, that's... >> It's a terabyte of data. (both conversing) >> It's a bit of a challenge. And if you snapshot it, you have two terabytes of data or you just snapshot the, like and get you to do a, okay, this is currently where we're at. So that's why the technology is evolving. How do we do source control management for data? How do we license data? How do we make sure that the data is unbiased, et cetera? So that's going more into the AI side of things. But at dealing with data in a declarative way in a containerized way, I think that's where currently a lot of innovation is happening. >> What do you mean by dealing with data in a declarative way? >> If I'm saying I run this experiment based on this data set and I'm running this other experiment based on this other data set, and I as the researcher don't care where the data is stored, I care that the data is accessible. And so I might declare, this is the process that I put on my data, like a data processing pipeline. These are the steps that it's going through. And eventually it will have gone through this process and I can work with my data. Pretty much like applying the concept of pipelines through data. Like you have these data pipelines and then now you have cube flow pipelines as one solution to apply the pipeline concept, to well, managing your data. >> Given the stateless nature of containers, is that an impediment to HPC adoption because of the very large data sets that are typically involved? >> I think it is if you have terabytes of data. Just, you have to get it to the place where the computation will happen, right? And just uploading that into the cloud is already a challenge. If you have the data sitting there on a supercomputer and maybe it was sitting there for two years, you probably don't care. And typically a lot of universities the researchers don't necessarily pay for the compute time they use. Like, this is also... At least in Ghent that's the case, it's centrally funded, which means, the researchers don't have to worry about the cost, they just get access to the supercomputer. If they need two terabytes of data, they get that space and they can park it on the system for years, no problem. If they need 200 terabytes of data, that's absolutely fine. >> But the university cares about the cost? >> The university cares about the cost, but they want to enable the researchers to do the research that they want to do. >> Right. >> And we always tell researchers don't feel constrained about things like compute power, storage space. If you're doing smaller research, because you're feeling constrained, you have to tell us, and we will just expand our storage system and buy a new cluster. >> Paul: Wonderful. >> So you, to enable your research. >> It's a nice environment to be in. I think this might be a Jevons paradox problem, you give researchers this capability you might, you're going to see some amazing things. Well, now the people are snapshoting, one, two, three, four, five, different versions of a one terabytes of data. It's a good problem to have, and I hope to have you back on theCUBE, talking about how Red Hat and Ghent have solved those problems. Thank you so much for joining theCUBE. From Valencia, Spain, I'm Keith Townsend along with Paul Gillon. And you're watching theCUBE, the leader in high tech coverage. (upbeat music)

>> Welcome back to AnsibleFest. I'm Matthew Jones, I'm the architect of the Ansible Automation Platform. And today I want to talk to you a little bit about what we've got coming in 2021, and some of the things that we're working on for the future. Today, I really want to cover some of the work that we're doing on scale and flexibility, and how we're going to focus on that for the next year. I also want to talk about how we're going to help you grow and manage and use your content on the Automation platform. And then finally, I want to look a little bit beyond the automation platform itself. So, last year we introduced Ansible Content Collections. Earlier this year, we introduced the Ansible Automation Hub on Red Hat Cloud. And yesterday you heard Richard mentioned on private automation hub that's coming later this year. And automation hub, Ansible tower, this is really what the automation platform means for us. It's bringing together that content, with the ability to execute and run and manage that content, that's really important. And so what we really want to do, is we want to help you bring Red Hat and partner content that you trust together with community content from galaxy that you may need, and bring this together with content that you develop for yourself, your roles, your collections, the automation that you actually do. And we want to give you control over that content and help you curate that content and build a community around your automation. We want to focus on a seamless experience with this automation from Ansible Tower and from Automation Hub for the automation platform itself, and make it accessible to the automation and infrastructure that you're managing. Now that we've talked about content a little bit, I want to talk about how you run Ansible. Today an Ansible Tower, use virtual environments to manage the actual execution of Ansible, and virtual environments are okay, but they have some drawbacks. Primarily they're not very portable. It's difficult to manage dependencies and the version of Ansible. Sometimes those dependencies conflict with the other systems that are on the infrastructure itself, even Ansible Tower. So what we've done is created a new system that we call execution environments. Execution environments are container-based. And what we're doing is bringing the flexibility and portability of containers to these Ansible execution environments. And the goal really is portability. And we want to be able to leverage the tools that the community develops as well as the tools that Red Hat provides to be able to produce these container images and use them effectively. At Ansible we've developed a tool called Ansible Builder. Ansible builder will let you bring content collections together with the version of Ansible and Red Hats base container image so that you can put together your own images for execution environments. And you'll be able to host these on your own private registry infrastructure. If you don't already have a container registry solution, Automation Hub itself provides that registry. The idea here is that, unlike today where your virtual environments and your production execution environments diverge a little bit from what your developers, your content developers and your automation developers experience, we want to give you the same experience between your production environments and your development environments, all the way through your test and validation workloads. Red Hat's also going to provide some prebuilt execution environments. We want to have some continuity between the experience that you have today on the Ansible tower and what you'll have next year, once we bring execution environments into production. We want you to be able to trust the Ansible, the version of Ansible that's running on your execution environments, and that you have the content that you expect. At the same time, we're going to provide a version of the execution environment, that's just the base execution environment. All it has is Ansible. This will let you take those using Ansible builder, take the collections that you've developed, that you need in your automation and combine them without having to bring in things that you don't need, or that you don't want in your automation and build them together into a very opinionated, container image. If you're interested in execution environments and you want to know how these are built and how you'll use them, we actually have them available for you to use today. Shane McDonald and Adam Miller are giving a talk later with a walk through how to build execution environments and how you'll use them. You can use this to make sure that you're ready for execution environments coming to the automation platform next year. Now that we've talked about how we build execution environments, I want to talk about how execution runs in your infrastructure. So today when you deploy Ansible tower, you're deploying a monolithic web application. Your execution capability is tied up into how you actually deploy Ansible tower. This makes scaling Ansible tower and your automation workloads difficult, and everything has to be co-located together in the same data center. Isolated nodes solve this a little bit, but they bring about their own sort of opinionated challenges in setting up SSH and having direct connectivity between the control nodes and the execution nodes themselves. We want to make this more flexible and easier to use. And so one of the things that we've created over the last year and that we've been working on over the last year is something that we call receptor. Receptor is an overlay network that's an Automation Mesh. And the goal here is to separate the execution capability of your Ansible content from the control plane capability, where you manage the web infrastructure, the users, the role-based access control. We want to draw a line between those. We want you to be able to deploy execution environments anywhere. Chris Wright earlier today mentioned Edge. Well Edge Cloud, we want you to be able to manage data centers anywhere in the world, and you can do this with the Automation Mesh,. The Automation Mesh connects your control plane with those execution nodes, anywhere in the world. Another thing that the Automation Mesh brings is, we're going to be able to draw the lines between the control plane themselves and each Automation Mesh node. This means that if you have an outage or a problem on your network and on your infrastructure, if you can draw a line between the control plane itself and the node that needs to execute, the sensible work, the Automation Mesh can route around problems. The Automation Mesh in the way it's deployed, also allows this to fit closer with ingress and egress policies that you have between your infrastructure. It doesn't matter which direction the Automation Mesh itself connects in. Once the connection is established, automation will be able to flow from the control systems to the execution nodes and get responses back. Now, this all works together with automation of the content collections that we mentioned earlier, the execution environments that we were just talking about and your container registries. All of these work together with these Automation Mesh nodes. They're very lightweight and very simple systems. This means you can scale up and scale down execution capacity as your needs increase or decrease. You don't need to keep around a lot of extra capacity just in case you automate more, just because you're not sure when your execution capacity needs will increase and decrease. This fits into an automated system for scaling your infrastructure and scaling your execution capacity. Now that we've talked about the content that you use to manage, and how that execution is performed and where that execution is performed. I want to look a little bit beyond the actual automation platform itself. And specifically, I want to talk about how the automation platform works with OpenShift and Kubernetes. Now we have an existing installer for Ansible tower that we'll deploy to OpenShift Kubernetes, and we support OpenShift and Kubernetes as a first-class system for deploying Ansible tower. But I mentioned automation hub and Ansible tower as this is what the automation platform is for us. So we want to take that installer and replace it with an operator-based full life cycle approach to deploying and managing the automation platform on OpenShift. This operator will be available in OperatorHub. So there's no need to manage complex YAML files that represent the deployment. Since it's available in OperatorHub, you have one place that you can go to manage deployments, upgrades, backup and restore. And all of this work seamlessly with the container groups feature that we introduced last year. But I want to take this a little bit beyond just deploying and upgrading the automation platform from the operator. We want to look at what other capabilities that we can get out of those operators. So beyond just deploying and upgrading, we're also creating a resource operators and CRDs that will allow other systems running in OpenShift or Kubernetes to directly manage resources within the automation platform. Anything from triggering jobs and getting the status of jobs back, we want to enable that capability if you're using OpenShift and Kubernetes. The first place we're starting with this, is Red Hats Advanced Cluster Management system. Advanced Cluster Management brings together the ability to manage OpenShift and Kubernetes clusters to install them and manage them, as well as applications and products in managing the life cycle of those across your clusters. So what we really want to do, is give you the ability to connect traditional and container-based workloads together. You're already using the Ansible automation platform to manage workloads with Ansible. When using Advanced Cluster Management and OpenShift and Kubernetes, now you have a full system. You can manage across clouds across clusters, anywhere in the world. And this sort of brings me back to one of the areas of focuses for us. Our goal is complete end-to-end automation. We want to connect your people, your domains and the processes. We want to help you deliver for you and your customers by expanding the capabilities of the Ansible automation platform. And we want to make this a seamless experience to both curate content, control the content for your organization, and run the content and run Ansible itself using the full suite of the Ansible automation platform. So the Advanced Cluster management team is giving a talk later where you'll actually be able to see Advanced cluster Management and the Ansible automation platform working together. Don't forget to check out Adam and Shane's talk on execution environments, how those are built and how you can use those. Thank you for coming to AnsibleFest, and we'll see you next time.