>> Announcer: From around the globe, it's The Cube. With digital coverage of Ansible Fest 2020 brought to you by Red Hat. >> Hello, welcome back to The Cube's coverage of Ansible Fest 2020. This is The Cube. Cube Virtual. I'm your host, John Furrier with The Cube and Silicon Angle. Two great guests here. Two engineers and architects. Michael McCarthy who is a architect at Delivery Engineering, who's giving a talk with Gamesys and Jurgen Grech who's a technical architect for the platform engineering team at Gamesys. Gentlemen, welcome to The Cube, thanks for coming on. >> Hello. >> Nice to see you. >> Coming in from London, coming in from Malta, you guys are doing a lot of engineering. You're a customer of Ansible, want to get into some of the cool things you're doing obviously Kubernetes automation, platform engineering, this is what everyone's working on right now that's going to be positioned for the future. Before we get started though, tell me a little bit about what Gamesys does and you guys' role. Michael, we'll start with you. >> Sure, so we're a gaming operator, we run multiple bingo-led and casino-led gaming websites, some of them are B2B, some are B2C. I think we've been doing it now for probably 14 or 15 years at least. I've been there for 12 and a half of those. So we essentially run gaming websites where people come and play their favorite games. >> And what's your role there? What do you do? >> So I'm in the operation side of things, I used to be a developer for 12 or so years. We make sure that everything's kind of up and running, we keep the systems running. My team in particular focuses on the speed of delivery for developers so we're constantly looking at, how long has it taken to get things in front of the customers, can we make it faster, can we make it easier, can we put cool stuff out there quicker? So it's a kind of platformy type role that I do, and I enjoy it a lot, so it's good. >> Jurgen you're platform engineering that sounds deep. >> Yes. >> Which is your role? (laughing) >> Well, I've been with Gamesys also for eight and a half years now. I hold the position of technical architect at the moment within this platform engineering group which is mostly tasked with all things ops related. I am responsible for designing, implementing and validating strategies for continuous deployment, whilst always ensuring high availability on both production and pre-production systems. I'm also responsible for the design and implementation of automated dynamic environment to support the needs of the development teams and also collaborating with other architects, especially those on the development floors in order to optimize the deployment and operational strategies for both existing and new types of services alike. >> Awesome, thanks for sharing that. Good, good context. Well, I mean, you don't have to be a rocket scientist to figure out that when you talk about gaming it's uptime and a high availability is critical. You know, having people, being the login you got to have the right data strategies, it can't be down, right. (laughs) It's a critical app. People are not going to enjoy it if they're not at, so I can see how scale's huge. Can you guys talk about how Ansible fits in because automation's been the theme here, you guys have been having a journey with automation. What's been your automation solution with Ansible? >> I'll go Michael. >> Yeah sure. >> So, basically back in July 2014, we started to look at Ansible to replace those commonly used, day to day, best scripts, which our ops team use to execute and which could lead to some human error. That was our main original goal of using Ansible at the time. At the time was our infrastructure looked considerably different. Definitely much, much smaller than the current private cloud footprint. And as I said, as early adopters within the operations team it was imperative for us to automate as much as possible. Those repetitive tasks, which involved the execution of various scripts and were prone to human error. Since then however, aware Ansible usage, it worked quickly. Since 2014, we went through two major infrastructure overhauls and automation using Ansible was always at the heart of each of those overhauls. In fact, our latest private cloud which is based on OpenStack is completely built from the ground up using Ansible code. So this includes the provision and co-visual machines, our entire networking stacks, so switches, routers, firewall, the SDN which OpenStack is built up on, our internal DNS system. Basically all you need to have a fully functional private cloud. At Gamesys we also have some workloads running in two different public clouds. And even in this case, we are running against the build code to set up all the required infrastructure components. Again, since we were fairly new adopters at the time of this technology, without all of those Ansible code, using the original as the case, cover now this has worked considerably and with enhancements of litigated modules polished public cloud, we've made the code look much cleaner, readable and ad approved. >> You made some great progress. Michael, you want to weigh in on this? Any thoughts on? >> Yeah, I think it's kind of, I mean, adding to what Jurgen said I think it's kind of everywhere. So, you know, you mentioned, you mentioned high availability, you mentioned kind of uptime, you know, imagine the people that operate the infra, the people who get called out and they're working 24 seven, you know, a lot of the things that they would do, the kind of run books they would use to, you know, restart something they're Ansible as well. So it's the deployment scripts, it's the kind of scripts that keep things running, it's the stuff that spins up the environments as Jurgen said. I've noticed a lot on the development side where, you know, we look at continuous delivery, people are running their own build servers. A lot of the scripting that people do, which, you know you'd imagine, might be done with say Bash, I think I've seen a lot of Ansible being used there amongst developers, I guess. Yeah, it's got an easy learning curve. It's all of those modules. A lot of the scripting around CD I think is Ansible. It plays quite nicely, you know, URI module and file modules and yeah, I think it's kind of everywhere I think. It's quite pervasive. >> Once again I said, when to get something going. Good, it's awesome. >> Yeah. Automation get great success. So it's been a big theme of Ansible Fest 2020 automation collectors, et cetera. But the question I have for you guys as customers, is how large of an IT estate were you looking to automate and where was the most imperative places to automate first? >> The most imperative items we wanted to automate first as I said, were those operational day to day tasks handled by our network operations team. Our estate is massive. So we are running our infrastructure across five different data centers around the world, thousands of virtual machines, hundreds of network components. So we, we deal with customers all around the world. So our point of presence is spread out around the world as well. And you can't really handle such kind of size without some sort of automation. And Ansible fit the bill perfectly, in my opinion. >> And so your goal is to automate the entire landscape. Are you there now? Where are you on that progress? >> I would say we're at a very advanced stage in that process. Since 2014 we've made huge strides. All of our most recent private cloud setups as I said, have been built from the ground up using Ansible. And I would say a good 90% plus of our operational tasks are handled using some kind of Ansible playbook. >> Yeah, that makes total sense. Michael you brought up the, you start early in people's, it spreads. Those are my words, but you were saying that. What kind of systems do people tend to start with at Ansible? And what's, where's that first sticky moment where it lands and expands and which teams jump on it first? Is it the developers? Is it more the IT? Take us through some of the how this all gets started and how it spreads. >> I think in the, the first time I remember using it was probably I think 2014, 2015. And it was what Jurgen mentioned. I was on the Dev side and we wanted a way to have consistency in how we deployed. We wanted to be able to deploy the exact same way, you know into earlier environments, into Dev environments as we did in staging and production. And, you know, someone kind of found Ansible and then someone in operations kind of saw it and they were happy with it and they felt comfortable using the, kind of getting up to speed. And I think it was hard to know where it really started first, but you sort of looked around and every team, every team kind of had it. So, you know, who actually started I'm not sure, but it's all over the place. >> He did. (laughs) >> Yeah. I think, you know, where people start with it first it probably depends if you're on the ops or the dev side, I think on the dev side you know, we're encouraging people to own their own deployment playbooks you know, you're responsible for the deployment of your system to production. Obviously you've got the network operations the not group sort of doing it for you, but you know, your first exposure is probably going to be writing a playbook to deploy your app or maybe it's around some build tooling, spinning up your own build environment but that's something you'll be doing. I know with Ansible and it's especially around this point of stuff because everything's in git, there's that collaboration which I never saw, obviously I saw people chatting over kind of slack in teams but in terms of being able to sort of raise PR's having developers raise PR's, having operations comment on them the same the other way around, that's been a massive change which I think has come from using Ansible. >> The collaboration piece is huge. And I think it's one of those things early on out of all the Ansible friends that I know that use it and customers and in the company product was just good. It just word of mouth, spreads it around and be like, this is workable, saves a lot of time and it's a pain point remover. Also enables some things to happen with now automation, but now it's mature. Right? So Jurgen I got to ask you in the maturation of all this automation you're talking about scale, you mentioned it. OpenStack, you guys got the private clouds, people use it for public cloud, I now see Red Hat has a angle on that. But when you think about the current modern state of the art today, you can't go anywhere without talking about Kubernetes. >> Yup. >> Kubernetes has really emerged on the scene to manage these clusters but yet it's just getting started. You have a lot of experience with Ansible and Kubernetes. Can you share your journey with Kubernetes and Ansible, and what's your reaction to that? >> Yes, so back in June 2016 Gamesys was developing a new gaming platform which was stood on now Kubernetes. Kubernetes at the time was fairly new to many at an enterprise level with only a handful of production systems online. So we were tasked to assess how we're going to bring Kubernetes into production. So we first, we identified the requirements to set up a production grade cluster and given our experience with Ansible, we embarked on a journey to automate the installation process. Again using Ansible this would ensure that all the required installation and configuration parameters as Michael mentioned, we are committing it, the code is shared with all the respective development teams for ease of collaboration and feedback. And we decided to logically divide our code into two. And we said, we're going to have an installation code in order to provide Kubernetes as a service. So this basically installs Docker onto every worker node. It installs cube lit, all the master playing components of Kubernetes installs core DNS, the container storage interface, and they full blown and cluster monitoring stack. Then we also had our configuration code which basically sets up name spaces, it labels nodes for specific uses at certain security policies according to the cluster use case and creates all the required role based access configurations. This need to split the code in two came about really with the growing adoption of Kubernetes because at the inception stage we only had the one team which had a requirement to use Kubernetes. However, with various teams getting on board each required their own flavor with their particular unique configurations. This is of course well managed quite easily to reduce of different Ansible inventories. And it's all integrated now within Ansible Tower with different unique drop templates to install and configure the Kubernetes clusters. We started as I said with just one pre-production or staging cluster in 2010 16. Today we manage 42 different Kubernetes clusters including six which are in production. >> What problems >> So, as I mentioned earlier >> I got to ask you 'cause Kubernetes certainly when it came out, I mean, that was a big fan boy of that. I was promoting Kubernetes from the beginning. I saw it as a really great opportunity to bring things together with containers. It turns out that developers love it for that reason. What, so getting your hands on is great, but as you moved it in to practice, what problems did it solve for you? >> So using Ansible, definitely solve the problem of ensuring that all of our 42 clusters across all the different data centers are running the same configuration. So they're running the same version. They're running the same security policies. They're running the same name space, according to the type. Each team has a similar deployment token. And it's very, very convenient to roll out changes and upgrades especially when all of our code has been integrated with Ansible Tower through a simple user interface click. >> How's Ansible Tower working for you? Is that going well? Ansible Tower? >> Eh, I would say so, yes. Most of our code now is integrated with Ansible Tower. It's allowed us to also share some of the tasks with a wider group of people. Within Peg we are the guardians of the production environments really. However, we share the responsibility of staging environments with the respective development teams, who primarily those environments. So as such, through the use of Ansible Tower we've managed to also securely and consistently share the same way how they can install and upgrade these clusters themselves without our involvement. >> Thank you. Michael you're giving, oh sorry go ahead. Go ahead Jurgen. >> Sorry is no no. >> Michael, you're giving a presentation breakout session at Ansible Fest. Can you give us a sneak peek >> Yup. >> Of what you're going to talk about? >> Yeah sure. So we, I said we've been using Tower for a long time. We've been using it since 2015 I think. Think we've probably made some mistakes along the way, I guess, or we've learned a lot of stuff from how we started then to now. So what it does is it follows this sort of timeline of how we started, why there was this big move to making an effort to put all of our deployment playbooks in Ansible. Why you would go to Tower over and above Ansible itself. It talks about our early interactions with quite an old version of Tower and now version two, things that we struggled with, then we saw version three came out there was loads and loads of really good stuff in version three. And it's really about kind of how we've used the new features, how it's worked out for us. It's kind of about what Gamesys have done with Tower but I think it's probably applicable to everyone and anyone that uses Tower I think will, they'll probably come across the same things, how do I scale it for multiple teams? How do I give teams the ownership to kind of own their own playbooks? How do I automate Tower itself? It talks about that. Sort of check pointing every few years about where we'd got to and what was going well and what was going less well. So, and a bit of a look forward to, what's going to come next with Tower. So we're constantly keeping up to date and we've got kind of roadmap for where we want to go. >> What's interesting about you guys is you think about look at OpenStack and then how Cloud came on the scene and Private Cloud has emerged with hybrid and obviously public, you guys are right on the wave of all this large scale stuff and your gaming app really kind of highlights that. And you've been through the paces with Ansible. So I guess my question, and you've got a lot of scar tissue and you got success to show for it too, a lot of great stuff. What advice would you give people who are now getting on the new wave, the bigger wave that's coming which is more users, more scale, more features more automation, microservices are coming around the corner. As long as I get more scale. What advice would you give someone who's coming on board with Ansible for the first time? >> I think there was, you were talking before about Kubernetes and it was so where we were, I think we'd got into containers kind of relatively early. And we were deploying Docker and we had some pretty big, kind of scary playbooks and they managed low balances and deployed Docker containers. And it was always interesting thinking how is this all going to change when Kubernetes comes along? And I think that's been really smooth. I think there's a really nice Ansible module that's just called gates. And I think it's really simple actually, it simplified a lot of the playbooks. And I think that the technologies can coexist quite happily. I don't think you have to feel like Kubernetes is going to change all of the investment you've made into Ansible. Even if you go down the route of Kubernetes operators, you can write them in Ansible. So I still think it's a very relevant tool even with Kubernetes being so kind of prevalent. >> Jurgen what's your thoughts on folks getting in now, who want to jump in and take advantage of the automation, all the cool stuff with Ansible? What advice would you give them? >> Yes, I would definitely recommend to look at their infrastructure set ups as they would look at their code. So break it down into small manageable components, start small, build your roles, make sure to build your roles properly for each of that small component. And then definitely look at Ansible Tower as a way to visualize and control the execution of your code. Make sure you're running it with the proper security policies with the proper credentials and all, they're not, of course so break anything which is at the production level. >> Michael McCarthy, Jurgen Grech two great engineers at Gamesys. Congratulations on your success and love to unpack the infrastructure and the scale you have and certainly automation, great success path. And it's going to get easier. I mean, that's what everyone's saying, it's going to get easier. Thanks for coming on. I appreciate the conversation. Thank you very much. >> Thank you, welcome >> Thank you, take care. Bye bye. >> I'm John Furrier with The Cube here in Palo Alto California. We're virtual, The Cube virtual for Ansible Fest 2020 virtual. Thank you for watching. (upbeat music)