from around the globe it's thecube with coverage of kubecon and cloudnativecon north america 2020 virtual brought to you by red hat the cloud native computing foundation and ecosystem partners hey welcome back everybody jeff frick here with thecube coming to you from our palo alto studios with our ongoing coverage of kubecon cloud nativecon north america 2020 virtual it's virtual like everything else that we're doing in 2020 we're really excited by our next guest we're going to dive into a company that you probably know a little bit on the surface but probably don't know a lot of the stuff that's going on behind the surface so we're really excited to have our next guest he is jai chakrabarti he is the director of engineering for core infrastructure at spotify jai great to see you great to be here with you today so as a as a long-standing uh spotify fan and and customer and premium customer and family playing customer just so there's no question i'm a big fan the infrastructure to deliver what i want to hear basically any sound any song from the entire world it seems like i don't know what the actual uh percentage of every published song you guys have you know kind of at my fingertips searchable available now to listen to is an amazing accomplishment i can't imagine how big and significant and complicated the infrastructure you guys must be managing and and not only that but kind of the meteoric growth over the last several years so first off just talk a little bit about spotify scale how you guys think about it is there some things that you can share to help people really understand you know some of the some of the big iron that's behind giving me the songs i want to hear absolutely and thank you for the opportunity to let me talk about this so it's a as you say it's a pretty mammoth project to be able to deliver just about any song that's in the world or now any podcast that you might want to listen to to hundreds of millions of fans and also enable creators to be able to share their content with the consumers who are interested in consuming that content so some of the metrics that go behind us are we have thousands of microservices running in production we were one of the early adopters of microservices at scale and continued to build on that foundation with early entrants to dockerize services and now of course largely on kubernetes we also have thousands of data pipelines hundreds of uh websites as well as micro app features and we're doing about 20 000 deployments a day to give you kind of a scale of how fast things are changing and for us speed is a great virtue as we're testing out features doing ab tests and trying to roll out the next best thing for the audio network it's amazing and i'm and i'm curious in terms of execution on the business side i mean clearly you're in many many countries you know you're global are all the licensing agreements for the music different by country are you just like super micromanaging um you know kind of the the revenue streams and the licensing by geo or is is that just as complex as it feels like it might be or is there some some simplicity or some scale that you can bring to uh to bring a little bit of of clarification there yeah so that is an area of complexity as well um so you know licensing across the broad set of content that we have as well as the number of publishers and creators that we have to make sure that everything is well accounted for is also kind of a source of complexity in our organizational makeup and then and then the the piece that i don't think a lot of people know is you guys are huge consumers and contributors back to open source and clearly we're here at q con cloud native con you've talked already about kubernetes and containers but i wonder before we get into some of the specifics if you can talk about philosophically the role of open source and why you know you guys are such a big open source company versus kind of back in the old days when you would have a lot of proprietary technology that you would try to develop and keep in-house as part of the as part of the secret sauce yeah thank you for that question so philosophically we are big proponents of open source we believe in giving back to the community we believe that when we as a community come together to solve these problems at scale the end result is much better than if we were to try it alone if any one company were to try it alone so some of the projects that we've contributed or invested a lot of time in are envoy for example which we use to power our perimeter at spotify or kubernetes which we use for deployment purposes as many companies do but there are also a number of other open source projects that we're committing to so for example with cloud bigtable we have produced an auto scaler that's now fairly widely used to be able to manage costs better with cloud bigtable we've also invested in a open source time series database called heroic to manage millions of data points for a metrics platform and scales so those are just a few examples but philosophically we believe this isn't something that we want to do alone and we want to leverage and do this together with the community right another one that you didn't mention there but you've talked about i want to dig into is backstage and as you mentioned you have a lot of developer teams working on a lot of projects like i saw a statistic maybe in github of the number of of github projects you guys are working on it's a it's a lot so what is backstage all about give us the story there yeah so at spotify we have almost somewhere around 500 engineering teams and so you can think about backstage as kind of like a central nervous system to be able to help engineers interface across the wide landscape that is spotify's engineering ecosystems so if you're an engineer you can go into backstage and you can manage your services your data pipelines your micro features you can see what other teams are doing what the organizational structure is you can get recommendations and insights on your tech health so you can see where you might need to invest more time and get some recommendations on how to get back to the blessed stock so it's really a one-stop developer portal that engineers spend the bulk of their time in today we open sourced it uh earlier this year and we've been absolutely thrilled with the response we've gotten thus far a number of companies have already started using it and contributing back so we've seen you know a lot of contributions coming back to backstage which is of course one of the ideas to be able to get some of the great ideas uh on backstage so we're really excited about that and specifically within backstage something that my team has just released into the open is a product called cost insights so one of the problems that we were dealing with at spotify is how do we sustainably look at cloud costs but do it in a way that isn't like a compliance exercise isn't a focus on traditional top top down cost controls but really taps into developers innate desire to work on optimization because all of us who come from an engineering background know that optimization is fun at the same time premature optimization is the root of all evil as the saying goes and so what we've done within our cost insights product and backstage is really try to find a good balance between engineering love for optimization and letting people know what are the areas where cloud spend really matters so if making an investment here isn't going to move the needle for us we let people know that this isn't worth your time to worry about so let me unpack you touch on a couple things first off you talked about it gives you an assessment of your engineering health so does that mean that it's kind of uh compliance within a standard is that looking for i guess not quite red flags yet but yellow flags of things that that are known potential issues down the road is it you know tapping into maybe higher cost services or microservices versus less that maybe there's a less expensive way so so how do you define health and how do you you know keep track of people getting away from health and then you know steering them back to being more healthy yeah that's a great question so we have this concept at spotify called golden state which is a reflection of how far away are you from all of the blessed frameworks libraries that we recommend to engineers and the way we think about golden state is there ought to be clear value adds to going to a new service a new library version and so the way we try to express it is unless of course there's a kind of a direct security concern and there aren't really too many ways to get around that but we really tried to preserve engineering autonomy and say if you go to this new framework for example you're going to save this much time on average so the recommendations that you'll find there are going to be highly specific so for example if you adopt uh you know an auto scaler for bigtable you're going to save this much time and spend this much less that's in general how we phrase these things okay and then on the cost insights i mean clearly when a dev is working on a new feature or new uh you know experimenting maybe with a bunch of new features and you're you're setting up multiple a b testing this and that are they are they not really working worrying about cost at the front end of that or is really kind of the cost optimization and you mentioned you know don't optimize too early does that come kind of after the fact and after you've you know moved some new things into production they have potential and now we do maybe a second order kind of analysis of the appropriateness of that feature because i imagine if they're just if you're just trying to come up with new features and exploring and trying new things not really worrying about the you're not worrying about the cloud bill right you're just trying to get some feature functionality and make sure you don't have too many bugs and make sure you're going to get some good client value and some new customer experience yeah yeah no and and we agree with that perspective so we think about the world in terms of startup scale-ups and mature businesses at spotify so there are a lot of teams who are experimenting with new ideas that fall into the startup category and by and large they are not going to be worrying about costs that being said we as infrastructure teams have the notice on us to think about how do we provide shared services and frameworks that abstract away a lot of these questions around how do you properly manage your costs right so that that is on us as infrastructure teams but really our perspective is for startups to move as quickly as they can and really if that's an idea that's viable and you get to what we call the scale-up stage or you get to the mature business stage where it really is a core part of our business then that's where you know you might start to get some nudges or recommendations and cost insights so interesting so i'd love to you know your background you came from financial services and trading where clearly speed matters accuracy matters you know that that's i mean basically financial services is is a software game at this stage of the game and it's a speed game and i saw another interesting uh video getting ready for this i think it was with gustav soderstrom talking about the competitive advantage of the early days really being speed and speed to return a result and speed to start that stream and it just struck me very much like you know the early days of google which was that was their whole speed thing and they even told you how fast you got a return on your search when you're thinking about optimizing now with the huge suite of features and functionalities that you have how do you think about speed is it still speed number one how is kind of the priority changed and what are some of the design priorities that when things go from experiment to start to be into the scale realm and hopefully be successful in production that that need to be thought about and potentially rank ordered um in in the proper way yeah yeah that's it's a great question and so you know i'll just refer to daniel x quote around this which is we aim to fail faster than anyone else and so for us as a company and with our growth trajectory and investing in the areas that we are looking to invest into it's still absolutely critical that we move fast that we get the ideas of the startup phase out to be vetted and validated if we can go to the next phase to the scale-up phase so i see that just as important today if not more than when i first joined spotify uh you know over four years ago at this point and regarding financial services um there are certainly you know touch points in terms of the amount of data that we're processing and the scale of technology that it requires to process that kind of data but one of the things that i really love about spotify of course is that we get to move fast which is sometimes of course going to be a lot more difficult when you're talking about the financial service arena and various uh compliance bodies that are overseeing any changes that you might make yeah you guys are you guys were running a little bit ahead of the regs i think which is pretty typical uh in the music business napster was running a little bit ahead of the regs and you know then we saw the evolution with the itunes and then you know you guys really really nailing the streaming service really for the first time and and opening up this new con consumption bottle and i wonder if you could talk about you know kind of keeping the customer experience first and making sure that that's a positive thing i can't help but think of of the netflix experience where they spend so much time on people's interaction with the application to to get them to try new things a recommendation engine such an important piece of the of the puzzle and i think what you guys have really nailed is the discovery piece because it's one thing to be able to quickly access a favorite song and be able to listen to it but everyone loves discovery right and discovery is kind of an interesting and interesting process and you guys have taken a really scientific approach in terms of cataloging music and and different attributes of music and then using those to help drive the recommendation engine i wonder if you can share you know kind of your thoughts in terms of being you know kind of ultimately driven by the customer experience and their interaction with the application and these things called you know music or podcast which is such a such a a a very personal thing to interact with yeah so from the perspective of core infrastructure you know it's spotify our goal is to really enable the scale in which we are processing the amount of audio content that goes through our system and so podcast of course is a new category that wasn't there when i originally joined spotify but it's really to provide a platform so these experiments can be done seamlessly so we can have different ways of looking at discovery looking at user segmentation and being able to come up with new ways that are going to be compelling to our customers so that's very exciting and fulfilling for us to be able to provide that platform by which our sister teams can iterate very quickly knowing that they have the guard rails uh which you know in our on-premise days at times was a struggle and where we're in a very different place now yeah so last question before i let you go we're at cubecon cloudnativecon um and and it's just an interesting thing that i always think about when you're managing engineering teams that are heavily open source participants and you know it's such a big piece now of of a lot of engineers motivation to be active participants in open source and to and to show their work to others outside the company but at the same time they have to get company work done so i just wonder if you could share your perspective of how do you manage open source contributions how do you keep them you know working on company projects but also make sure you allocate time and priorities to open source contributions because that is a really important piece of the motivation for a lot of engineers it's not just working for the company and getting paid at the india at the end of every two weeks yeah it's a key motivation as you say and it's key to our recruiting strategy and also how we think about retaining engineers and spotify so there are different mechanisms that we use and there's a lot of focus that's modified on coming up with development plans for engineers that actually make sense um so you know i would say that all the way from the oft quoted 20 time is something that you might hear at spotify where you have engineers who are working on open source 20 of the time or you might see a variety of customized customized options depending on who the engineer is where they want to grow and really i think the key here is providing the right support structures so even if you have the time are you getting the mentorship are you getting the right kind of support system so you know how to connect with the community and so you have other like-minded people who are bouncing ideas and you don't feel like you're doing it yourself so that's something that i feel really excited about that we've grown those support structures over the last few years eyes have also been very intentional about giving engineers time to work on open source and you give them as much as 20 i'd never heard that before yeah in some cases some i mean if that is what where an engineer really wants to focus and grow there are a number of folks at spotify who are spending up to 20 of their time on open source wow that's amazing that that is a uh that's a it's just it's such a great commitment for the company to the engineer if that's their priority and then everyone's going to benefit from it both the engineer the company as well as the community so really a forward-looking you know point of view to take that long-term view versus the you know maybe we should only give them 10 we're losing 10 of their time working on a project so that is super super progressive and i'm sure you must be seeing great roi on it or you wouldn't continue to be such huge proponents of open source and such huge contributors back so that's that's a great story yeah terrific i mean you know we we want those contributions to be in line with where we're growing as a company and we see a lot of opportunities uh where that is happening so like envoy or kubernetes um just to name a couple of examples where folks have devoted time in those areas well thanks for uh thanks for sharing some of the the story behind the scenes you know again household name what what a tremendous success story and and and uh you know i'm a movie customer so i'm definitely a customer though no no doubt about it so uh thank you for your contributions congrats to the team and uh and really loved the story of how you guys are contributing back and and doing a lot more than just making great music available to us all and a great channel for uh for creators to get their stuff out there so thanks again thanks so much for your time i really appreciate it all right he's jai i'm jeff you're watching the cube's continuing coverage of kubecon cloud nativecon north america 2020 thanks for watching we'll see you next [Music] time you

(upbeat electronic music) >> Announcer: Live from Vancouver, Canada, it's theCUBE, covering OpenStack Summit North America 2018. Brought to you by Red Hat, the OpenStack Foundation, and its ecosystem partners. >> Welcome back to The Cube's coverage of OpenStack Summit 2018 in Vancouver. I'm Stu Miniman with my co-host John Troyer. Happy to welcome back a company we've spoken to a few times at events, Paddy Power Betfair. First time guest coming to us from across the pond, Dave Buckley who is the automation engineer with Paddy Power Betfair, thanks for joining us. >> Thank you for having me. >> Alright, so first of all, you've been to a couple summits and we've talked to Paddy Power about OpenStack. Before we get into your specific implementation, tell us about your experience here this week and any compare, contrast to previous years. >> Yeah so I'm very lucky, I got to come to the previous two summits in North America. I guess what I've enjoyed this week, it's kind of a slight tilt towards, it's away from being purely OpenStack, kind of towards this open infrastructure kind of thing, 'cause like I said, especially last year in Boston, Q and NEs was becoming a big thing. Yeah, and kind of, the OpenStack Foundation becoming kind of more, not that it wasn't before, but more community-based and being part of the ecosystem. So, yeah, I think it's been quite interesting seeing that. >> Not to put words in your mouth but, it was even, the last year or two, it's more aware of some of the complimentary things and adding pieces. You know, we had, one of the interviews we did this week was person who's the SIC lead for the Kubernetes stuff, that sits under another Foundation, things like that. Yeah, exactly. It's been quite interesting this week, I guess, sort the Kata Container project, which wasn't something I'd been aware of before Monday morning basically. I remember we were sitting in the keynotes, and they were like, you can have this container-like thing which has all the speed of a container, but it's as secure as a BM. And you're thinking, how, how is that even possible? So I've really enjoyed, I got to go to one of the sessions yesterday, one of the technical introductions on that. >> Yeah, I always love, there's certain things where, okay, this is what I'm going to do with my schedule, and turns into, this got announced, or I didn't know about this, and you knew, blow up my schedule, let me change everything else. Yeah, exactly, I think you always, you can't, you have to be flexible, right? Adaptable, and as the week goes on you just go to what you think is interesting. >> John: So Dave, you and your company have been working with OpenStack for quite a while. >> Dave: Yeah. >> And you obviously run a system that needs to be stable. Right, needs to, you take care of betting and people's money. >> Dave: Exactly. >> So that needs to be solid. But I understand you recently went though an upgrade and have some experiences talking about that? Can you talk a little bit about where you are with your OpenStack implementation and that sort of migration? >> Sure. So, I guess it's about three years ago, it was Betfair at the time, so this was before the merger of the two companies. So Betfair started using OpenStack, and I think it was actually the last time the summit was here, in Vancouver. So a couple of my colleagues who were kind of the technical leads at the time. Steve Armstrong and Steve Perera, they flew out here, to kind of get a feel for OpenStack, what it was, talk to people who'd had experiences with it. I actually think that conference back then was very informative of what the platform today now looks like. So some of the conversations they had there with people like New Age Networks and Arista, which we used for the switching, but conversations they had there kind of ended up being now what we're using in production. I guess over the past couple of years, so the big thing that happened obviously was this merger between Paddy Power and Betfair, following that they had an exercise which they called the single customer platform, which is annoyingly, for a sys-admin guy, kind of like me, they, it's always been abbreviated to SCP, but you have to ignore that. So that was to kind of consolidate and integrate the Paddy Power and Betfair co-bases and put it on a single platform, which was our OpenStack and Nuage platform. So that kind of completed in January this year, so that's live, so basically the Paddy Power sports book has an entirely new website, all running on OpenStack. A lot quicker and more efficient then the previous version. So that's been a real success. And as part of that, I should say that stability is really vital, so kind of in our business. If the site is down we don't make any money, and if it happens during a big sporting event you have a big problem. >> Do you have a metric around that? What a minute or an hour of down time would be? >> So I guess it always depends, so the nature of our traffic is very spikey. So obviously when you have a big, it's on a Saturday in Europe, the football, soccer, maybe I should say, is like a very big deal. >> We have a global audience, football's okay. >> I'll stick with football then. >> We were all watching the royal wedding. >> I don't want to talk about that. The football, if you, we just get peak traffic on that day. And, even within the year, there's a thing called the Grand National, which is a big event in the UK, big horse racing, I guess like the Kentucky Derby. It's kind of when we get our maximum traffic in the year. Yeah, you always need to be prepared for that. So one of the things as you mentioned, we kind of look into upgrade OpenStack from Kilo to Newton. So we've been on Kilo from the start. We're using Red Hat's distribution of OpenStack, so what Red Hat offer is this, they have like every three releases I think it is? They have this long release life-cycle. So that's kind of the reason we're going to Newton, cause we have kind of the, then the support will go to 2021. [Stu] - But if I remember, it's Red Hat the OpenStack Platform 10. >> Dave: Yeah. >> And 13 is going to be queened as their next one that's going to be released. >> Exactly, so I think they just announced that this week, right? So I think at some point in the next year or two we'd be going to queens. >> How do you determine when you make that jump and anything around the upgrade process, you know, good and bad that you could share. >> Dave: Yeah, so I guess going from, we were overdue an upgrade in this case, Kilos, you know, pretty old now. What we're lucky that we can do is because we have Nuage, it's like an external SDM provider, so the entire data plane is controlled by Nuage, and you can kind of plug as many OpenStacks as you like really into Nuage, and you offload all the networking to Nuage. So what's that's allowed us to do is basically we'd have had a lot of trouble if we'd had to do an in place upgrade, so I've actually been to one of the groups this week, quite a lot of people were talking about upgrades and just like all the nightmares it's caused. I know it's getting better as like the releases come out, but what we were able to do is kind of building new, an entirely new OpenStack cloud on the side of, so we've kind of turned it kind of an immutable OpenStack, so your OSB 7 cloud is there, we built this new OSB 10. But they're both circ into the same networking, so the same Nuage SDN. And the way our developers deploy their applications, I guess you want to see this in more detail, we've done presentations at these summits in the past, but kind of in short, every deployment we do immutable deployments as well, so for every deployment we'll create a new subnet within Nuage, and kind of do rolling update of your VMs that are on that new subnet into like a VIP which is kind of where the constant is, so all the traffic's come in to that VIP then you just flip things in and out below it when you do a deployment, so what that basically means is from a developers point of view, when they're migrating from OSB 7 to OSB 10 they'll essentially spin up new networks and new VMs in OSB 10 and that deployment pipeline will kind of just seamlessly, everything else will stay the same because the networking doesn't change. So we don't have to have any downtime on the data plane or the control plane. Which is really beneficial for us 'cause the way, I guess this is I'll just describe the way developers do deployments like we rely heavily on the OpenStack API being available. You pay a cost in that you, so you need extra hardware to do that I guess, but yeah we found it is something that's worked for us. >> John: Anything else with the networking and specifically that you all are running, the load balancing or resiliency that you need to have for your apps? >> Dave: Yeah so one of the things was, so it's kind of another problem there were trying to solve with this whole project, this new OpenStack platform is that historically Betfair, as it was at the time, had always run out of a single data-center. But we had another site, but it was mainly kind of a development environments right in there. So the company thought why don't we just have, we should just have both DCs for resiliency, try and run things in like an active-active configuration. Which is fine for external customer facing applications where we've had an external load balance server that can point traffic between the two DCs. But then the question is what do you do with internal apps? So this is what led us to use Avi Networks, which is kind of a cloud native load balancing technology, so we've been using to provide like GSLB internal laps, so basically we'll load balance traffic between the two data-centers so it gets deployed within your OpenStack environment, has a really neat integration with Nuage, the Nuage SDN layer, and will resolve you to whichever data-center is appropriate at that time. So if you have a full data-center outtage, you should be able to go "Okay, point stuff over there". >> John: So it makes you and the networking team or the IT team into the heroes not the villains, you're usually the people saying "No" or "We can't do that". >> I guess so, I guess so yeah you're probably right. It's cool technology though. I guess that we're very lucky and that we're given the opportunity by the people at the company to experiment with new things, so even though we're about stability but we're also about trying to push things forward in terms of what technology to use. >> Stu: Dave I'm curious how kind of the hybrid or multi-cloud type of environments fit into what you're doing today, give us the update there. >> Dave: Yeah so that's something very in our radar at the moment I guess it's, yeah it's what everybody's doing, looking to how you can have this hybrid cloud model. So I think, going back three years again, at that time, being like an online betting company, it's a highly regulated business and only at that point it was really possible to kind of put some of this stuff into the public cloud, it seems like things have come a long way, so it's something we're looking at at the moment, we're evaluating different solutions, different vendors like the Googles, AWSs, and seeing or even like some OpenStack public clouds and seeing maybe how could we migrate some workloads out into the public cloud, how do we want to that, to give us more resiliency, and also as I was saying about our spiky traffic, it just makes a lot of sense to be able to say burst out into whichever public cloud vendor on a Saturday when the football's on to deal with that peak load. So it's something we're very much looking at at the moment. But yeah no formal decisions as of yet. Unless they've done something while I've been away. >> John: With containers here at the show, lots of different threads right? Containers, Edge, the OpenDev track, things like that. Anything else, we've talked about Kata, anything else that came up that was interesting here that you just watch Kubernetes and container track as well? >> Dave: So I guess in terms of containers it's, sitting in the Keynotes on Monday you would, if you weren't watching if you were just listening, you probably wouldn't know you were at an OpenStack Summit right since there's as much Kubernetes container stuff as there is OpenStack. It's interesting so we've kind of been doing... Again, similar to the public cloud conversation, it's something that's very relevant to us at the moment, we've done kind of a few proof-of-concept ideas, evaluating different solutions, so we have like running a Cube cluster ourself, obviously we have a strong relationship with Red Hat that we've kind of explored to using OpenShift maybe, and then come the networking layer you can integrate with Nuage which would be really cool for us so it'll allow us to do kind of the all the networking, access control mechanisms as we do for our virtual machines. And again this is also something in the whole public cloud conversation is well if wanted to containers in the public cloud as well like you have all the different offerings, would we want to run our own, in like an AWS or something? Or maybe go to someone like Google where you have that supported self-service model I suppose. But yeah at the moment it's kind of at those stages so I think Steve did a presentation on the Kubernetes stuff like a PCO we done at the last Summit. But yeah still at the moment still want to make some firm decisions about which direction we're going to go but a lot of the developers a very keen for this and obviously for guys like us we all know the value of it so I think at the moment because we had that focus on stability we should now have a period of time where we're able to kind of look at all this stuff a bit more, hopefully get some container solutions into production which would be awesome. >> Stu: Dave Buckley we really appreciate you giving us the update, love to be able to do some of those longitudinal case studies as to where you've been where you're going, what you're thinking about. Be sure to check out thecube.net, you can actually search for Patty Power Betfair, see some of those previous interviews from Dave's peers. Loads more interviews there as well as all the shows we're going to be at in the future where hope you come by and say "Hi". For John Troyer I'm Stu Miniman, thanks so much for watching theCUBE. >> (electro-dance music) >> (soft piano)