(upbeat electronic music) >> Announcer: Live from Vancouver, Canada, it's theCUBE, covering OpenStack Summit North America 2018. Brought to you by Red Hat, the OpenStack Foundation, and its ecosystem partners. >> Welcome back to The Cube's coverage of OpenStack Summit 2018 in Vancouver. I'm Stu Miniman with my co-host John Troyer. Happy to welcome back a company we've spoken to a few times at events, Paddy Power Betfair. First time guest coming to us from across the pond, Dave Buckley who is the automation engineer with Paddy Power Betfair, thanks for joining us. >> Thank you for having me. >> Alright, so first of all, you've been to a couple summits and we've talked to Paddy Power about OpenStack. Before we get into your specific implementation, tell us about your experience here this week and any compare, contrast to previous years. >> Yeah so I'm very lucky, I got to come to the previous two summits in North America. I guess what I've enjoyed this week, it's kind of a slight tilt towards, it's away from being purely OpenStack, kind of towards this open infrastructure kind of thing, 'cause like I said, especially last year in Boston, Q and NEs was becoming a big thing. Yeah, and kind of, the OpenStack Foundation becoming kind of more, not that it wasn't before, but more community-based and being part of the ecosystem. So, yeah, I think it's been quite interesting seeing that. >> Not to put words in your mouth but, it was even, the last year or two, it's more aware of some of the complimentary things and adding pieces. You know, we had, one of the interviews we did this week was person who's the SIC lead for the Kubernetes stuff, that sits under another Foundation, things like that. Yeah, exactly. It's been quite interesting this week, I guess, sort the Kata Container project, which wasn't something I'd been aware of before Monday morning basically. I remember we were sitting in the keynotes, and they were like, you can have this container-like thing which has all the speed of a container, but it's as secure as a BM. And you're thinking, how, how is that even possible? So I've really enjoyed, I got to go to one of the sessions yesterday, one of the technical introductions on that. >> Yeah, I always love, there's certain things where, okay, this is what I'm going to do with my schedule, and turns into, this got announced, or I didn't know about this, and you knew, blow up my schedule, let me change everything else. Yeah, exactly, I think you always, you can't, you have to be flexible, right? Adaptable, and as the week goes on you just go to what you think is interesting. >> John: So Dave, you and your company have been working with OpenStack for quite a while. >> Dave: Yeah. >> And you obviously run a system that needs to be stable. Right, needs to, you take care of betting and people's money. >> Dave: Exactly. >> So that needs to be solid. But I understand you recently went though an upgrade and have some experiences talking about that? Can you talk a little bit about where you are with your OpenStack implementation and that sort of migration? >> Sure. So, I guess it's about three years ago, it was Betfair at the time, so this was before the merger of the two companies. So Betfair started using OpenStack, and I think it was actually the last time the summit was here, in Vancouver. So a couple of my colleagues who were kind of the technical leads at the time. Steve Armstrong and Steve Perera, they flew out here, to kind of get a feel for OpenStack, what it was, talk to people who'd had experiences with it. I actually think that conference back then was very informative of what the platform today now looks like. So some of the conversations they had there with people like New Age Networks and Arista, which we used for the switching, but conversations they had there kind of ended up being now what we're using in production. I guess over the past couple of years, so the big thing that happened obviously was this merger between Paddy Power and Betfair, following that they had an exercise which they called the single customer platform, which is annoyingly, for a sys-admin guy, kind of like me, they, it's always been abbreviated to SCP, but you have to ignore that. So that was to kind of consolidate and integrate the Paddy Power and Betfair co-bases and put it on a single platform, which was our OpenStack and Nuage platform. So that kind of completed in January this year, so that's live, so basically the Paddy Power sports book has an entirely new website, all running on OpenStack. A lot quicker and more efficient then the previous version. So that's been a real success. And as part of that, I should say that stability is really vital, so kind of in our business. If the site is down we don't make any money, and if it happens during a big sporting event you have a big problem. >> Do you have a metric around that? What a minute or an hour of down time would be? >> So I guess it always depends, so the nature of our traffic is very spikey. So obviously when you have a big, it's on a Saturday in Europe, the football, soccer, maybe I should say, is like a very big deal. >> We have a global audience, football's okay. >> I'll stick with football then. >> We were all watching the royal wedding. >> I don't want to talk about that. The football, if you, we just get peak traffic on that day. And, even within the year, there's a thing called the Grand National, which is a big event in the UK, big horse racing, I guess like the Kentucky Derby. It's kind of when we get our maximum traffic in the year. Yeah, you always need to be prepared for that. So one of the things as you mentioned, we kind of look into upgrade OpenStack from Kilo to Newton. So we've been on Kilo from the start. We're using Red Hat's distribution of OpenStack, so what Red Hat offer is this, they have like every three releases I think it is? They have this long release life-cycle. So that's kind of the reason we're going to Newton, cause we have kind of the, then the support will go to 2021. [Stu] - But if I remember, it's Red Hat the OpenStack Platform 10. >> Dave: Yeah. >> And 13 is going to be queened as their next one that's going to be released. >> Exactly, so I think they just announced that this week, right? So I think at some point in the next year or two we'd be going to queens. >> How do you determine when you make that jump and anything around the upgrade process, you know, good and bad that you could share. >> Dave: Yeah, so I guess going from, we were overdue an upgrade in this case, Kilos, you know, pretty old now. What we're lucky that we can do is because we have Nuage, it's like an external SDM provider, so the entire data plane is controlled by Nuage, and you can kind of plug as many OpenStacks as you like really into Nuage, and you offload all the networking to Nuage. So what's that's allowed us to do is basically we'd have had a lot of trouble if we'd had to do an in place upgrade, so I've actually been to one of the groups this week, quite a lot of people were talking about upgrades and just like all the nightmares it's caused. I know it's getting better as like the releases come out, but what we were able to do is kind of building new, an entirely new OpenStack cloud on the side of, so we've kind of turned it kind of an immutable OpenStack, so your OSB 7 cloud is there, we built this new OSB 10. But they're both circ into the same networking, so the same Nuage SDN. And the way our developers deploy their applications, I guess you want to see this in more detail, we've done presentations at these summits in the past, but kind of in short, every deployment we do immutable deployments as well, so for every deployment we'll create a new subnet within Nuage, and kind of do rolling update of your VMs that are on that new subnet into like a VIP which is kind of where the constant is, so all the traffic's come in to that VIP then you just flip things in and out below it when you do a deployment, so what that basically means is from a developers point of view, when they're migrating from OSB 7 to OSB 10 they'll essentially spin up new networks and new VMs in OSB 10 and that deployment pipeline will kind of just seamlessly, everything else will stay the same because the networking doesn't change. So we don't have to have any downtime on the data plane or the control plane. Which is really beneficial for us 'cause the way, I guess this is I'll just describe the way developers do deployments like we rely heavily on the OpenStack API being available. You pay a cost in that you, so you need extra hardware to do that I guess, but yeah we found it is something that's worked for us. >> John: Anything else with the networking and specifically that you all are running, the load balancing or resiliency that you need to have for your apps? >> Dave: Yeah so one of the things was, so it's kind of another problem there were trying to solve with this whole project, this new OpenStack platform is that historically Betfair, as it was at the time, had always run out of a single data-center. But we had another site, but it was mainly kind of a development environments right in there. So the company thought why don't we just have, we should just have both DCs for resiliency, try and run things in like an active-active configuration. Which is fine for external customer facing applications where we've had an external load balance server that can point traffic between the two DCs. But then the question is what do you do with internal apps? So this is what led us to use Avi Networks, which is kind of a cloud native load balancing technology, so we've been using to provide like GSLB internal laps, so basically we'll load balance traffic between the two data-centers so it gets deployed within your OpenStack environment, has a really neat integration with Nuage, the Nuage SDN layer, and will resolve you to whichever data-center is appropriate at that time. So if you have a full data-center outtage, you should be able to go "Okay, point stuff over there". >> John: So it makes you and the networking team or the IT team into the heroes not the villains, you're usually the people saying "No" or "We can't do that". >> I guess so, I guess so yeah you're probably right. It's cool technology though. I guess that we're very lucky and that we're given the opportunity by the people at the company to experiment with new things, so even though we're about stability but we're also about trying to push things forward in terms of what technology to use. >> Stu: Dave I'm curious how kind of the hybrid or multi-cloud type of environments fit into what you're doing today, give us the update there. >> Dave: Yeah so that's something very in our radar at the moment I guess it's, yeah it's what everybody's doing, looking to how you can have this hybrid cloud model. So I think, going back three years again, at that time, being like an online betting company, it's a highly regulated business and only at that point it was really possible to kind of put some of this stuff into the public cloud, it seems like things have come a long way, so it's something we're looking at at the moment, we're evaluating different solutions, different vendors like the Googles, AWSs, and seeing or even like some OpenStack public clouds and seeing maybe how could we migrate some workloads out into the public cloud, how do we want to that, to give us more resiliency, and also as I was saying about our spiky traffic, it just makes a lot of sense to be able to say burst out into whichever public cloud vendor on a Saturday when the football's on to deal with that peak load. So it's something we're very much looking at at the moment. But yeah no formal decisions as of yet. Unless they've done something while I've been away. >> John: With containers here at the show, lots of different threads right? Containers, Edge, the OpenDev track, things like that. Anything else, we've talked about Kata, anything else that came up that was interesting here that you just watch Kubernetes and container track as well? >> Dave: So I guess in terms of containers it's, sitting in the Keynotes on Monday you would, if you weren't watching if you were just listening, you probably wouldn't know you were at an OpenStack Summit right since there's as much Kubernetes container stuff as there is OpenStack. It's interesting so we've kind of been doing... Again, similar to the public cloud conversation, it's something that's very relevant to us at the moment, we've done kind of a few proof-of-concept ideas, evaluating different solutions, so we have like running a Cube cluster ourself, obviously we have a strong relationship with Red Hat that we've kind of explored to using OpenShift maybe, and then come the networking layer you can integrate with Nuage which would be really cool for us so it'll allow us to do kind of the all the networking, access control mechanisms as we do for our virtual machines. And again this is also something in the whole public cloud conversation is well if wanted to containers in the public cloud as well like you have all the different offerings, would we want to run our own, in like an AWS or something? Or maybe go to someone like Google where you have that supported self-service model I suppose. But yeah at the moment it's kind of at those stages so I think Steve did a presentation on the Kubernetes stuff like a PCO we done at the last Summit. But yeah still at the moment still want to make some firm decisions about which direction we're going to go but a lot of the developers a very keen for this and obviously for guys like us we all know the value of it so I think at the moment because we had that focus on stability we should now have a period of time where we're able to kind of look at all this stuff a bit more, hopefully get some container solutions into production which would be awesome. >> Stu: Dave Buckley we really appreciate you giving us the update, love to be able to do some of those longitudinal case studies as to where you've been where you're going, what you're thinking about. Be sure to check out thecube.net, you can actually search for Patty Power Betfair, see some of those previous interviews from Dave's peers. Loads more interviews there as well as all the shows we're going to be at in the future where hope you come by and say "Hi". For John Troyer I'm Stu Miniman, thanks so much for watching theCUBE. >> (electro-dance music) >> (soft piano)