>> From around the globe, it's theCUBE with coverage of KubeCon and CloudNativeCon Europe 2020, Virtual. Brought to you by Red Hat, The Cloud Native Computing Foundation and Ecosystem partners. >> Welcome back, I'm Stu Miniman and this is CUBEs coverage of KubeCon CloudNativeCon 2020 in Europe, the virtual edition and of course one of the things we love when we come to these conferences is to get to the actual practitioners, understanding how they're using the various technologies especially here at the CNCF show, so many projects, lots of things changing and really excited. We're going to talk about security in a slightly different way than we often do on theCUBE so happy to welcome to the program from Sera4 I have Jeff Klink who's the Vice President of Engineering and Cloud. Jeff, thanks so much for joining us. >> Thanks too, thanks for having me. >> All right so I teed you up there, give us if you could just a quick thumbnail on Sera4, what your company does and then your role there. >> Absolutely so we're a physical hardware product addressing the telco markets, utility space, all of those so we kind of differentiate herself as a Bluetooth lock for that higher end space, the highest security market where digital encryption is really an absolute must. So we have a few products including our physical lock here, this is a physical padlock, it is where door locks and controllers that all operate over the Bluetooth protocol and that people can just use simply through their mobile phones and operate at the enterprise level. >> Yeah, I'm guessing it's a little bit more expensive than the the padlock I have on my shed which is getting a little rusty and needs a little work but it probably not quite what I'm looking for but you have Cloud, you know, in your title so give us if you could a little bit you know, what the underlying technology that you're responsible for and you know, I understand you've rolled out Kubernetes over the last couple of years, kind of set us up with what were the challenges you were facing before you started using that? >> Absolutely so Stu We've grown over the last five years really as a company like in leaps and bounds and part of that has been the scalability concern and where we go with that, you know, originally starting in the virtual machine space and, you know, original some small customers in telco as we build up the locks and eventually we knew that scalability was really a concern for us, we needed to address that pretty quickly. So as we started to build out our data center space and in this market it's a bit different than your shed locks. Bluetooth locks are kind of everywhere now, they're in logistics, they're on your home and you actually see a lot of compromises these days actually happening on those kind of locks, the home security locks, they're not built for rattling and banging and all that kind of pieces that you would expect in a telco or utility market and in the nuclear space or so you really don't want to lock that, you know, when it's dropped or bang the boat immediately begins to kind of fall apart in your hands and two you're going to expect a different type of security much like you'd see in your SSH certificates, you know, a digital key certificate that arrives there. So in our as we grew up through that piece Kubernetes became a pretty big player for us to try to deal with some of the scale and also to try to deal with some of the sovereignty pieces you don't see in your shed locks. The data sovereignty meeting in your country or as close to you as possible to try to keep that data with the telco, with the utility and kind of in country or in continent with you as well. That was a big challenge for us right off the bat. >> Yeah, you know Jeff absolutely, I have some background from the telco space obviously, there's very rigorous certifications, there's lots of environments that I need to fit into. I want to poke at a word that you mentioned, scale. So scale means lots of things to lots of different people, this year at the KubeCon CloudNativeCon show, one of the scale pieces we're talking about is edge just getting to lots of different locations as opposed to when people first thought about, you know, scale of containers and the like, it was like, do I need to be like Google? Do I have to have that much a scale? Of course, there is only one Google and there's only a handful of companies that need that kind of scale, what was it from your standpoint, is it you know, the latency of all of these devices, is it you know, just the pure number of devices, the number of locations, what was what was the scale limiting factor that you were seeing? >> It's a bit of both in two things, one it was a scale as we brought new customers on, there were extra databases, there was extra identity services, you know, the more locks we sold and the more telcos we sold too suddenly what we started finding is that we needed all these virtual machines and sources in some way to tie them together and the natural piece to those is start to build shared services like SSO and single sign on was a huge driver for us of how do we unite these spaces where they may have maintenance technicians in that space that work for two different telcos. Hey, tower one is down could you please use this padlock on this gate and then this padlock on this cabinet in order to fix it. So that kind of scale immediately showed us, we started to see email addresses or other on two different places and say, well, it might need access into this carrier site because some other carrier has a equipment on that site as well. So the scale started to pick up pretty quickly as well as the space where they started to unite together in a way that we said, well, we kind of have to scale to parts, not only the individuals databases and servers and identity and the storage of their web service data but also we had to unite them in a way that was GDPR compliant and compliant with a bunch of other regulations to say, how do we get these pieces together. So that's where we kind of started to tick the boxes to say in North America, in Latin America, South America we need centralized services but we need some central tie back mechanism as well to start to deal with scale. And the scale came when it went from Let's sell 1000 locks to, by the way, the carrier wants 8000 locks in the next coming months. That's a real scalability concern right off the bat, especially when you start to think of all the people going along with those locks in space as well. So that's the that's the kind of first piece we had to address and single sign on was the head of that for us. >> Excellent, well you know, today when we talk about how do i do container orchestration Kubernetes of course, is the first word that comes to mind, can you bring us back though, how did you end up with Kubernetes, were there other solutions you you looked at when you made your decision? What were your kind of key criteria? How did you choose what partners and vendors you ended up working with? >> So the first piece was is that we all had a lot of VM backgrounds, we had some good DevOps backgrounds as well but nobody was yet into the the container space heavily and so what we looked at originally was Docker swarm, it became our desktop, our daily, our working environment so we knew we were working towards microservices but then immediately this problem emerged that reminded me of say 10, 15 years ago, HD DVD versus Blu-ray and I thought about it as simply as that, these two are fantastic technologies, they're kind of competing in this space, Docker Compose was huge, Docker Hub was growing and growing and we kind of said you got to kind of pick a bucket and go with it and figure out who has the best backing between them, you know from a security policy, from a usage and size and scalability perspective, we knew we would scale this pretty quickly so we started to look at the DevOps and the tooling set to say, scale up by one or scale up by 10, is it doable? Infrastructure as code as well, what could I codify against the best? And as we started looking at those Kubernetes took a pretty quick change for us and actually the first piece of tooling that we looked at was Rancher, we said well there's a lot to learn the Kubernetes space and the Rancher team, they were growing like crazy and they were actually really, really good inside some of their slack channels and some of their groups but they said, reach out, we'll help you even as a free tier, you know and kind of grow our trust in you and you know, vice versa and develop that relationship and so that was our first major relationship was with Rancher and that grew our love for Kubernetes because it took away that first edge of what am i staring at here, it looks like Docker swarm, they put a UI on it, they put some lipstick on it and really helped us get through that first hurdle a couple years ago. >> Well, it's a common pattern that we see in this ecosystem that you know, open source, you try it, you get comfortable with it, you get engaged and then when it makes sense to roll it into production and really start scaling out, that's when you can really formalize those relationships so bring us through the project if you will. You know, how many applications were you starting with? What was the timeline? How many people were involved? Were there, you know, the training or organizational changes, you know, bring us through under the first bits of the project. >> Sure, absolutely. So, like anything it was a series of VMs, we had some VM that were load balanced for databases in the back and protected, we had some manual firewalls through our cloud provider as well but that was kind of the edge of it. You had your web services, your database services and another tier segregated by firewalls, we were operating at a single DCs. As we started to expand into Europe from the North America, Latin America base and as well as Africa, we said this has got to kind of stop. We have a lot of Vms, a lot of machines and so a parallel effort went underway to actually develop some of the new microservices and at first glance was our proxies, our ingresses, our gateways and then our identity service and SSL would be that unifying factor. We honestly knew that moving to Kubernetes in small steps probably wasn't going to be an easy task for us but moving the majority of services over to Kubernetes and then leaving some legacy ones in VM was definitely the right approach for us because now we're dealing with ingressing around the world. Now we're dealing with security of the main core stacks, that was kind of our hardcore focus is to say, secure the stacks up front, ingress from everywhere in the world through like an Anycast Technology and then the gateways will handle that and proxy across the globe and we'll build up from there exactly as we did today. So that was kind of the key for us is that we did develop our micro services, our identity services for SSO, our gateways and then our web services were all developed in containers to start and then we started looking at complimentary pieces like email notification mechanisms, text notification, any of those that could be containerized later, which is dealt with a single one off restful services were moved at a later date. All right. >> So Jeff, yeah absolutely. What to understand, okay, we went through all this technology, we did all these various pieces, what does this mean to your your business projects? So you talked about I need to roll out 8000 devices, is that happening faster? Is it you know, what's the actual business impact of this technology that you've rolled out? >> So here's the key part and here's a differentiator for us is we have two major areas we differentiate in and the first one is asymmetric cryptography. We do own the patents for that one so we know our communication is secure, even when we're lying over Bluetooth. So that's kind of the biggest and foremost one is that how do we communicate with the locks on how do we ensure we can all the time. Two is offline access, some of the major players don't have offline access, which means you can download your keys and assign your keys, go off site do a site to a nuclear bunker wherever it may be and we communicate directly with the lock itself. Our core technology is in the embedded controllers in the lock so that's kind of our key piece and then the lock is a housing around it, it's the mechanical mechanism to it all. So knowing that we had offline technology really nailed down allowed us to do what many called the blue-green approach, which is we're going down for four hours, heads up everybody globally we really need to make this transition but the transition was easy to make with our players, you know, these enterprise spaces and we say we're moving to Kubernetes. It's something where it's kind of a badge of honor to them and they're saying these guys, you know, they really know what they're doing. They've got Kubernetes on the back end, some we needed to explain it to but as soon as they started to hear the words Docker and Kubernetes they just said, wow, this guys are serious about enterprise, we're serious about addressing it and not only that they're forefront of other technologies. I think that's part of our security plan, we use asymmetric encryption, we don't use the Bluetooth security protocol so every time that's compromised, we're not compromised and it's a badge of honor we were much alongside the Kubernetes. >> Alright, Jeff the thing that we're hearing from a lot of companies out there is that that transition that you're going through from VMs to containerization I heard you say that you've got a DevOps practice in there, there's some skill set challenges, there's some training pieces, there's often, you know, maybe a bump or two in the road, I'm sure your project went completely smoothly but what can you share about, you know, the personnel skill sets, any lessons learned along the way that might help others? >> There was a ton. Rancher took that first edge off of us, you know, cube-cuddle, get things up, get things going, RKE in the Rancher space so the Rancher Kubernetes engine, they were kind of that first piece to say how do I get this engine up and going and then I'll work back and take away some of the UI elements and do it myself, from scheduling and making sure that nodes came up to understanding a deployment versus a DaemonSet, that first UI as we moved from like a Docker swarm environment to the the Rancher environment was really kind of key for us to say, I know what these volumes are, I know the networking and I all know these pieces but I don't know how to put core DNS in and start to get them to connect and all of those aspects and so that's where the UI part really took over. We had guys that were good on DevOps, we had guys are like, hey how do I hook it up to a back end and when you have those UI, those clicks like your pod security policy on or off, it's incredible. You turn it on fine, turn on the pod security policy and then from there, we'll either use the UI or we'll go deeper as we get the skill sets to do that so it gave us some really good assurances right off the bat. There were some technologies we really had to learn fast, we had to learn the cube-cuddle command line, we had to learn Helm, new infrastructure pieces with Terraform as well, those are kind of like our back end now. Those are our repeatability aspects that we can kind of get going with. So those are kind of our cores now is it's a Rancher every day, it's cube-cuddle from our command lines to kind of do those, Terraform to make sure we're doing the same thing but those are all practices we, you know, we cut our teeth with Rancher, we looked at the configs that are generated and said, alright, that's actually pretty good configure, you know, maybe there's a team to tolerance or a tweak we could make there but we kind of work backwards that way to have them give us some best practices and then verify those. >> So the space you're in, you have companies that rely on what you do. Security is so important, if you talk about telecommunications, you know, many of the other environments they have, you know, rigid requirements. I want to get to your understanding from you, you're using some open source tools, you've been working with startups, one of your suppliers Rancher was just acquired by SUSE, how's that relationship between you know, this ecosystem? Is that something that is there any concerns from your end user clients and what are your own comfort level with the moves and changes that are happening? >> Having gone through acquisitions myself and knowing the SUSE team pretty well, I'd say actually it's a great thing to know that the startups are funded in a great source. It's great to hear internally, externally their marketing departments are growing but you never know if a startup is growing or not. Knowing this acquisitions taking place actually gives me a lot of security. The team there was healthy, they were growing all the time but sometimes that can just be a face on a company and just talking to the internals candidly as they've always done with us, it's been amazing. So I think that's a great part knowing that there's some great open source texts, Helm Kubernetes as well that have great backers towards them, it's nice to see part of the ecosystem getting back as well in a healthy way rather than a, you know, here's $10,000 Platinum sponsorship. To see them getting the backing from an open source company, I can't say enough for. >> All right, Jeff how about what's going forward from you, what projects you're looking at or what what additions to what you've already done are you looking at doing down the road? >> Absolutely. So the big thing for us is that we've expanded pretty dramatically across the world now. As we started to expand into South Africa, we've expanded into Asia as well so managing these things remotely has been great but we've also started to begin to see some latencies where we're, you know, heading back to our etcd clusters or we're starting to see little cracks and pieces here in some of our QA environment. So part of this is actually the introduction and we started looking into the fog and the edge compute. Security is one of these games where we try to hold the security as core and as tight as you can but trying to get them the best user experience especially in South Africa and serving them from either Europe or Asia, we're trying to move into those data centers and region as well, to provide the sovereignty, to provide the security but it's about latency as well. When I opened my phone to download my digital keys I want that to be quick, I want the administrators to assign quickly but also still giving them that aspect to say I could store this in the edge, I could keep it secure and I could make sure that you still have it, that's where it's a bit different than the standard web experience to say no problem let's put a PNG as close as possible to you to give you that experience, we're putting digital certificates and keys as close as possible to people as well so that's kind of our next generation of the devices as we upgrade these pieces. >> Yeah, there was a line that stuck with me a few years ago, if you look at edge computing, if you look at IoT, the security just surface area is just expanding by orders or magnitude so that just leaves, you know, big challenges that everyone needs to deal with. >> Exactly, yep. >> All right, give us the final word if you would, you know, final lessons learned, you know, you're talking to your peers here in the hallways, virtually of the show. Now that you've gone through all of this, is there anything that you say, boy I wish I had known this it would have been this good or I might have accelerated things or which things, hey I wish I pulled these people or done something a little bit differently. >> Yep, there's a couple actually a big parts right off the bat and one, we started with databases and containers, followed the advice of everyone out there either do managed services or on standalone boxes themselves. That was something we cut our teeth on over a period of time and we really struggled with it, those databases and containers they really perform as poorly as you think they might, you can't get the constraints on those guys, that's one of them. Two we are a global company so we operate in a lot of major geographies now and ETC has been a big deal for us. We tried to pull our ETC clusters farther apart for better resiliency, no matter how much we tweak and play with that thing, keep those things in a region, keep them in separate, I guess the right word would be availability zones, keep them make redundant as possible and protect those at all costs. As we expanded we thought our best strategy would do some geographical distribution, the layout that you have in your Kubernetes cluster as you go global for hub-and-spoke versus kind of centralized clusters and pods and pieces like that, look it over with a with an expert in Kubernetes, talk to them talk about latencies and measure that stuff regularly. That is stuff that kind of tore us apart early in proof of concept and something we had to learn from very quickly, whether it'll be hub-and-spoke and centralize ETC and control planes and then workers abroad or we could spread the ETC and control planes a little more, that's a strategy that needs to be played with if you're not just in North America, South America, Europe, Asia, those are my two biggest pieces because those are our big performance killers as well as discovering PSP, Pod Security Policies early. Get those in, lock it down, get your environments out of route out of, you know, Port 80 things like that on the security space, those are just your basic housecleaning items to make sure that your latency is low, your performances are high and your security's as tight as you can make it. >> Wonderful, well, Jeff thank you so much for sharing Sera4 for story, congratulations to you and your team and wish you the best luck going forward with your initiatives. >> Absolutely, thanks so much Stu. >> All right, thank you for watching. I'm Stu Miniman and thank you for watching theCUBE. (soft music)