from around the globe it's thecube with coverage of kubecon and cloudnativecon north america 2020 virtual brought to you by red hat the cloud native computing foundation and ecosystem partners hey welcome back everybody jeff frick here with thecube coming to you from our palo alto studios with our ongoing coverage of kubecon cloud nativecon north america 2020 virtual it's virtual like everything else that we're doing in 2020 we're really excited by our next guest we're going to dive into a company that you probably know a little bit on the surface but probably don't know a lot of the stuff that's going on behind the surface so we're really excited to have our next guest he is jai chakrabarti he is the director of engineering for core infrastructure at spotify jai great to see you great to be here with you today so as a as a long-standing uh spotify fan and and customer and premium customer and family playing customer just so there's no question i'm a big fan the infrastructure to deliver what i want to hear basically any sound any song from the entire world it seems like i don't know what the actual uh percentage of every published song you guys have you know kind of at my fingertips searchable available now to listen to is an amazing accomplishment i can't imagine how big and significant and complicated the infrastructure you guys must be managing and and not only that but kind of the meteoric growth over the last several years so first off just talk a little bit about spotify scale how you guys think about it is there some things that you can share to help people really understand you know some of the some of the big iron that's behind giving me the songs i want to hear absolutely and thank you for the opportunity to let me talk about this so it's a as you say it's a pretty mammoth project to be able to deliver just about any song that's in the world or now any podcast that you might want to listen to to hundreds of millions of fans and also enable creators to be able to share their content with the consumers who are interested in consuming that content so some of the metrics that go behind us are we have thousands of microservices running in production we were one of the early adopters of microservices at scale and continued to build on that foundation with early entrants to dockerize services and now of course largely on kubernetes we also have thousands of data pipelines hundreds of uh websites as well as micro app features and we're doing about 20 000 deployments a day to give you kind of a scale of how fast things are changing and for us speed is a great virtue as we're testing out features doing ab tests and trying to roll out the next best thing for the audio network it's amazing and i'm and i'm curious in terms of execution on the business side i mean clearly you're in many many countries you know you're global are all the licensing agreements for the music different by country are you just like super micromanaging um you know kind of the the revenue streams and the licensing by geo or is is that just as complex as it feels like it might be or is there some some simplicity or some scale that you can bring to uh to bring a little bit of of clarification there yeah so that is an area of complexity as well um so you know licensing across the broad set of content that we have as well as the number of publishers and creators that we have to make sure that everything is well accounted for is also kind of a source of complexity in our organizational makeup and then and then the the piece that i don't think a lot of people know is you guys are huge consumers and contributors back to open source and clearly we're here at q con cloud native con you've talked already about kubernetes and containers but i wonder before we get into some of the specifics if you can talk about philosophically the role of open source and why you know you guys are such a big open source company versus kind of back in the old days when you would have a lot of proprietary technology that you would try to develop and keep in-house as part of the as part of the secret sauce yeah thank you for that question so philosophically we are big proponents of open source we believe in giving back to the community we believe that when we as a community come together to solve these problems at scale the end result is much better than if we were to try it alone if any one company were to try it alone so some of the projects that we've contributed or invested a lot of time in are envoy for example which we use to power our perimeter at spotify or kubernetes which we use for deployment purposes as many companies do but there are also a number of other open source projects that we're committing to so for example with cloud bigtable we have produced an auto scaler that's now fairly widely used to be able to manage costs better with cloud bigtable we've also invested in a open source time series database called heroic to manage millions of data points for a metrics platform and scales so those are just a few examples but philosophically we believe this isn't something that we want to do alone and we want to leverage and do this together with the community right another one that you didn't mention there but you've talked about i want to dig into is backstage and as you mentioned you have a lot of developer teams working on a lot of projects like i saw a statistic maybe in github of the number of of github projects you guys are working on it's a it's a lot so what is backstage all about give us the story there yeah so at spotify we have almost somewhere around 500 engineering teams and so you can think about backstage as kind of like a central nervous system to be able to help engineers interface across the wide landscape that is spotify's engineering ecosystems so if you're an engineer you can go into backstage and you can manage your services your data pipelines your micro features you can see what other teams are doing what the organizational structure is you can get recommendations and insights on your tech health so you can see where you might need to invest more time and get some recommendations on how to get back to the blessed stock so it's really a one-stop developer portal that engineers spend the bulk of their time in today we open sourced it uh earlier this year and we've been absolutely thrilled with the response we've gotten thus far a number of companies have already started using it and contributing back so we've seen you know a lot of contributions coming back to backstage which is of course one of the ideas to be able to get some of the great ideas uh on backstage so we're really excited about that and specifically within backstage something that my team has just released into the open is a product called cost insights so one of the problems that we were dealing with at spotify is how do we sustainably look at cloud costs but do it in a way that isn't like a compliance exercise isn't a focus on traditional top top down cost controls but really taps into developers innate desire to work on optimization because all of us who come from an engineering background know that optimization is fun at the same time premature optimization is the root of all evil as the saying goes and so what we've done within our cost insights product and backstage is really try to find a good balance between engineering love for optimization and letting people know what are the areas where cloud spend really matters so if making an investment here isn't going to move the needle for us we let people know that this isn't worth your time to worry about so let me unpack you touch on a couple things first off you talked about it gives you an assessment of your engineering health so does that mean that it's kind of uh compliance within a standard is that looking for i guess not quite red flags yet but yellow flags of things that that are known potential issues down the road is it you know tapping into maybe higher cost services or microservices versus less that maybe there's a less expensive way so so how do you define health and how do you you know keep track of people getting away from health and then you know steering them back to being more healthy yeah that's a great question so we have this concept at spotify called golden state which is a reflection of how far away are you from all of the blessed frameworks libraries that we recommend to engineers and the way we think about golden state is there ought to be clear value adds to going to a new service a new library version and so the way we try to express it is unless of course there's a kind of a direct security concern and there aren't really too many ways to get around that but we really tried to preserve engineering autonomy and say if you go to this new framework for example you're going to save this much time on average so the recommendations that you'll find there are going to be highly specific so for example if you adopt uh you know an auto scaler for bigtable you're going to save this much time and spend this much less that's in general how we phrase these things okay and then on the cost insights i mean clearly when a dev is working on a new feature or new uh you know experimenting maybe with a bunch of new features and you're you're setting up multiple a b testing this and that are they are they not really working worrying about cost at the front end of that or is really kind of the cost optimization and you mentioned you know don't optimize too early does that come kind of after the fact and after you've you know moved some new things into production they have potential and now we do maybe a second order kind of analysis of the appropriateness of that feature because i imagine if they're just if you're just trying to come up with new features and exploring and trying new things not really worrying about the you're not worrying about the cloud bill right you're just trying to get some feature functionality and make sure you don't have too many bugs and make sure you're going to get some good client value and some new customer experience yeah yeah no and and we agree with that perspective so we think about the world in terms of startup scale-ups and mature businesses at spotify so there are a lot of teams who are experimenting with new ideas that fall into the startup category and by and large they are not going to be worrying about costs that being said we as infrastructure teams have the notice on us to think about how do we provide shared services and frameworks that abstract away a lot of these questions around how do you properly manage your costs right so that that is on us as infrastructure teams but really our perspective is for startups to move as quickly as they can and really if that's an idea that's viable and you get to what we call the scale-up stage or you get to the mature business stage where it really is a core part of our business then that's where you know you might start to get some nudges or recommendations and cost insights so interesting so i'd love to you know your background you came from financial services and trading where clearly speed matters accuracy matters you know that that's i mean basically financial services is is a software game at this stage of the game and it's a speed game and i saw another interesting uh video getting ready for this i think it was with gustav soderstrom talking about the competitive advantage of the early days really being speed and speed to return a result and speed to start that stream and it just struck me very much like you know the early days of google which was that was their whole speed thing and they even told you how fast you got a return on your search when you're thinking about optimizing now with the huge suite of features and functionalities that you have how do you think about speed is it still speed number one how is kind of the priority changed and what are some of the design priorities that when things go from experiment to start to be into the scale realm and hopefully be successful in production that that need to be thought about and potentially rank ordered um in in the proper way yeah yeah that's it's a great question and so you know i'll just refer to daniel x quote around this which is we aim to fail faster than anyone else and so for us as a company and with our growth trajectory and investing in the areas that we are looking to invest into it's still absolutely critical that we move fast that we get the ideas of the startup phase out to be vetted and validated if we can go to the next phase to the scale-up phase so i see that just as important today if not more than when i first joined spotify uh you know over four years ago at this point and regarding financial services um there are certainly you know touch points in terms of the amount of data that we're processing and the scale of technology that it requires to process that kind of data but one of the things that i really love about spotify of course is that we get to move fast which is sometimes of course going to be a lot more difficult when you're talking about the financial service arena and various uh compliance bodies that are overseeing any changes that you might make yeah you guys are you guys were running a little bit ahead of the regs i think which is pretty typical uh in the music business napster was running a little bit ahead of the regs and you know then we saw the evolution with the itunes and then you know you guys really really nailing the streaming service really for the first time and and opening up this new con consumption bottle and i wonder if you could talk about you know kind of keeping the customer experience first and making sure that that's a positive thing i can't help but think of of the netflix experience where they spend so much time on people's interaction with the application to to get them to try new things a recommendation engine such an important piece of the of the puzzle and i think what you guys have really nailed is the discovery piece because it's one thing to be able to quickly access a favorite song and be able to listen to it but everyone loves discovery right and discovery is kind of an interesting and interesting process and you guys have taken a really scientific approach in terms of cataloging music and and different attributes of music and then using those to help drive the recommendation engine i wonder if you can share you know kind of your thoughts in terms of being you know kind of ultimately driven by the customer experience and their interaction with the application and these things called you know music or podcast which is such a such a a a very personal thing to interact with yeah so from the perspective of core infrastructure you know it's spotify our goal is to really enable the scale in which we are processing the amount of audio content that goes through our system and so podcast of course is a new category that wasn't there when i originally joined spotify but it's really to provide a platform so these experiments can be done seamlessly so we can have different ways of looking at discovery looking at user segmentation and being able to come up with new ways that are going to be compelling to our customers so that's very exciting and fulfilling for us to be able to provide that platform by which our sister teams can iterate very quickly knowing that they have the guard rails uh which you know in our on-premise days at times was a struggle and where we're in a very different place now yeah so last question before i let you go we're at cubecon cloudnativecon um and and it's just an interesting thing that i always think about when you're managing engineering teams that are heavily open source participants and you know it's such a big piece now of of a lot of engineers motivation to be active participants in open source and to and to show their work to others outside the company but at the same time they have to get company work done so i just wonder if you could share your perspective of how do you manage open source contributions how do you keep them you know working on company projects but also make sure you allocate time and priorities to open source contributions because that is a really important piece of the motivation for a lot of engineers it's not just working for the company and getting paid at the india at the end of every two weeks yeah it's a key motivation as you say and it's key to our recruiting strategy and also how we think about retaining engineers and spotify so there are different mechanisms that we use and there's a lot of focus that's modified on coming up with development plans for engineers that actually make sense um so you know i would say that all the way from the oft quoted 20 time is something that you might hear at spotify where you have engineers who are working on open source 20 of the time or you might see a variety of customized customized options depending on who the engineer is where they want to grow and really i think the key here is providing the right support structures so even if you have the time are you getting the mentorship are you getting the right kind of support system so you know how to connect with the community and so you have other like-minded people who are bouncing ideas and you don't feel like you're doing it yourself so that's something that i feel really excited about that we've grown those support structures over the last few years eyes have also been very intentional about giving engineers time to work on open source and you give them as much as 20 i'd never heard that before yeah in some cases some i mean if that is what where an engineer really wants to focus and grow there are a number of folks at spotify who are spending up to 20 of their time on open source wow that's amazing that that is a uh that's a it's just it's such a great commitment for the company to the engineer if that's their priority and then everyone's going to benefit from it both the engineer the company as well as the community so really a forward-looking you know point of view to take that long-term view versus the you know maybe we should only give them 10 we're losing 10 of their time working on a project so that is super super progressive and i'm sure you must be seeing great roi on it or you wouldn't continue to be such huge proponents of open source and such huge contributors back so that's that's a great story yeah terrific i mean you know we we want those contributions to be in line with where we're growing as a company and we see a lot of opportunities uh where that is happening so like envoy or kubernetes um just to name a couple of examples where folks have devoted time in those areas well thanks for uh thanks for sharing some of the the story behind the scenes you know again household name what what a tremendous success story and and and uh you know i'm a movie customer so i'm definitely a customer though no no doubt about it so uh thank you for your contributions congrats to the team and uh and really loved the story of how you guys are contributing back and and doing a lot more than just making great music available to us all and a great channel for uh for creators to get their stuff out there so thanks again thanks so much for your time i really appreciate it all right he's jai i'm jeff you're watching the cube's continuing coverage of kubecon cloud nativecon north america 2020 thanks for watching we'll see you next [Music] time you