live from Orlando Florida it's the cube coverage conf 18 got to you by Splunk hey welcome back to Splunk kampf 18 conf 8 hashtag Splunk conf 18 my name is Dave Volante I'm here with my co-host a minimun you watching the cube the leader and live tech coverage there's two days of wall-to-wall coverage is our seventh year stew at conf we're seeing the evolution of Splunk from kind of analyzing log files to having deep business impact across the organization and doing more with data Jim Nichols is here is the DevOps manager in Improv odda healthcare company good to see it thanks for coming to the cube again thank you for having me thank you so tell us about M privada and then love the the title DevOps in the title we'll get into that sure first the company yep so in providers the healthcare IT security company and we provide health court healthcare organizations around the world with secure Identity Management multi-factor authentication and enable just ubiquitous access to whatever sort of medical systems that they need to get into and we really try to enable healthcare by establishing trust between the medical providers the patient's the data and do that all securely and seamlessly so that we're not Security's not a part of their workflow it's just in there and they don't have to think about it and they just get access to what they need when they need it so I hear yeah on your website trust between people technology and information reminds me a little bit of a certain software company that branding is all around us today that is there seems like there's a line up between what Splunk does in your company's mission oh they're there absolutely is and you know like Splunk in privada has a very strong on premises in the data center footprint and we're expanding that into the cloud and that's where most of my work is is kind of managing those cloud systems that kind of complement the on-premise appliance and we're looking at how that's going to move into the cloud and what that means and it's very similar to like what Splunk is done with Splunk enterprise and now moving into Splunk cloud and we're actually a customer's point cloud everything that we do that we could possibly do is out in the cloud not in the data center in you you've got DevOps two new titles maybe bring us inside you know what that means that improv odda you usually think about you know moving fast things are changing all the time it's themes that we heard in the keynote this morning so explain that a little bit yeah so the way the DevOps model that we follow at improv odd is really like kind of a consultant model where we've got a small team of a very senior very expert DevOps folks and they kind of get assigned out to the agile teams and they're a team member that gets planned into the Sprint's plan and what we're going to be dealing and really kind of make sure that those deployment events or the DevOps work that we need to do is planned in as part of the normal development work and that consultancy model is really good in regards to Splunk because we run the Splunk infrastructure we do all the training we do some of the basic dashboard work and make sure that no matter what the team products onshore offshore wherever they are we're all looking at data exactly the same way exact same dashboards and it really kind of forces the knowledge to get shared throughout the organization across products and how we think about things and so Splunk you know DevOps isn't like a tool or a thing or whatever but Splunk is definitely a great like enabling forcing function to make sure that we are sharing metrics how the system works what we're learning on and all that stuff in a really consistent way so you know the t-shirt met tricks I've seen that I have what do you think that means oh so it's like the same old same old man metrics so huh what does that mean to you guys you have new metrics do you have a sort of new set of KPIs that you're using ourselves so I think the metrics part is that it's maybe 10 years ago the IT industry figured out how to get every single metric about CPU memory disk ram and all the tool there are a lot of different tools for doing it you know Splunk zabbix data dog others I don't know if it's okay to talk about other products or whatever but you know when you get like a CPU alert that goes off all right the CPU usage is 92% is that good or is that bad it sounds kind of high and you get that alert you look at that CPU chart and it's like there's no context there's no information and you know you might be designing your system to run at 90% if it's doing some batch processing or something so it's like metrics it's like you need to get the alert you need to know what's going on but you really need to like get the insight into what it is and that's why a lot of this stuff that they show this morning at the keynote was really exciting where you've got the metrics in one place the logs in one place it's all in one place so you get that alert and you can look at it and then see what else is going on without having to like jump into a bunch of different systems and how about DevOps your DevOps in the title what is how do you guys look at DevOps what is DevOps to you and where did it come from and where is it going I think that I've been doing DevOps my entire career since I got out of college and I came out of WPI and was studying like performance evaluation and it's like how do you measure systems get the insight how do you make sure they're running efficiently and I think that what I was kind of doing on the performance engineering side kind of intersected with like the agile movement and folks get into agile development teams and trying to integrate that knowledge and the metrics and how you're gonna run it in production into that sort of product building process so I feel like I've been doing DevOps for a long time and called it different things over the years you know for for us at improv Adi it's really about enabling our developers to deliver functionality to our customers as fast and as safely as possible so you know we're in the healthcare industry and you know the the systems that we build and integrate and support support life right like these are doctors that are using these systems they have to work a hundred percent all the time and that adds some interesting wrinkles where you wouldn't really think about doing continuous deployment for the system that you know somebody's going to get logged into to get into their medical records you might want to be able to move that quickly if you need to if there's an emergency bug fix but the level of safety and testing that we need to put in before it actually gets into production that's really where we spend a lot of our time in DevOps is making sure that that's a fish but that's fast and then when it goes from going from like a test environment into production if it takes an hour for office is not that big of a deal we're doing like you know multi week to week release cycles or even longer and so as far as like DevOps a lot of the movement has been around like continuous delivery and deployment and we kind of use that to optimize like the test build and debug cycle and that way when we know when we get to production that's going to go smoothly and that there aren't going to be any unanticipated how do security fit into this conversation sometimes you know the the buzzword term you know dev sac ops is you know how to how to Splunk in your practice look at security well so you know where a security company you know you know and we wouldn't really ever call anything dev sack offs because security is ingrained in a part of every single thing that we do walking into the building every day when we badge in I think about it our security people like is the building's secure all the way into like what we're ending up doing in the system so obviously Splunk is a huge supporter of that so we've got audit trail information on all the systems and we can know not only what you are system administrators and DevOps users are doing but like what docker is doing what commands it runs and really get at a very very low level of detail and we literally have everything that ever happens on those systems is audited and we've built a whole set of alerts around things that we know about things that we think might be a problem and we use kind of our expertise in the healthcare security space and then apply that to all our cloud systems so it's like we never have a team called dev sack ops it's like it's it's just what we do it's the first most important thing that we think about every day is security so that's why it's a little bit different for us but we like some of the ideas and I've you know we've started doing some work around automated security testing on the application code you know running like static analysis dynamic analysis integrating web scanning tools into our CI CD pipelines so that it just makes it that much easier you know and not wait till the end before you ship it or whatever we have it right in the development process what's the regime for your organization you know the classic development and operations throw it over the fence and okay DevOps brings those together but you still got a spectrum of skills and presumably you've got people on you know some kind of maturity model where you've got sort of newer folks maybe guys coming out of college like you were several years ago and you're training them and sort of you're one unified team at the same time you you might have some degrees of specialization so what have you found is the right regime for the DevOps team well I think the consultant model that we've established works really well and we've got a very senior DevOps person that's on the agile team and they may do some of the really tricky bits but once we get out of the part that only us as DevOps can do we really try to get the developers to do it so a lot of that's like Splunk training how do you build a dashboard here's maybe a simple example dashboard now you do the next panel that sort of thing to try to level everybody up and get everybody on the same page you know turned in terms of this divide between like Devon Ops when I actually joined and provided DevOps was NIT it was managed as part of like our SAS management offering along with like a lot of the other applications that IT managed and one of the very first things that our senior vice-president did was like they get to be in development they can't Oregon is a we were working together we're all on the same teams we're all doing all that stuff but just mentally organizationally get rid of the divide put them in engineering and report to the VP of engineering just like the developers development managers and architects and that's the way we've just get rid of any organizational or thought divide between the between the groups Jim you mention alerts just now and we've heard a few times you get alerts and you know I imagined the beeper in the old days now you get an alert on your mobile phone where are we in terms of being able to take action on those alerts have the machines take action for us is that an objective that you have is that just too damn scary your thoughts yeah so my first my first impression is that it's a little scary we do have some problems that occur with some frequency right so losing an Amazon ec2 instance happens you know 10 times out of 100 instances in the cloud on a given month so there's certain types of those failures that we've automated around just because you have to as a part of just doing business in the cloud so why do the Amazon like auto scaling groups all that stuff we've got a couple of you know issues that happen that we want to just resolve faster and repair faster they don't impact customer experience or user experience but we just want to get on top of those sooner so we've started to automate some of the very thin small carefully controlled controlled use cases so that if the alert were to go up spurious lis I know it's not gonna then take down a system that was running and finding good false positives exactly so only were places where false positives can be tolerated is where we're looking to do that yeah you don't want to take the humans out of the equation just yet or maybe ever for some of the simple things we we have and we can and we will but some of the complicated things it's like just stop and look at it and think about it for 90 seconds and then make the action we're to come up with how to program that 90 seconds of thought is like maybe talk about it be complete about it off oh this way okay let me explain it to somebody a second time and make sure it's right and then go and do a quake like just philosophically that's where I have to get a sheen to do that so Jim you're wearing the revolution a word shirt my understanding in privada is now one of two two-time Award winners if I got it right you're a commander Award winner maybe you could explain what that means and what it means to you and your company sure so the commander award is really about getting you know other folks in your organization using Splunk looking you know either looking at a dashboard at a report or digging down into the data and you know so why I won the award was really around like our use of docker containers so it was really important to me that developers people in DevOps people and support don't really have like a strong like network operations function but those types of folks that they're all looking at the exact same thing all the exact same tools all the exact same data so kind of as part of that mission it's just I hold trainings I hold office hours I've got one of my DevOps folks down here today or at the conference to then kind of spread the Splunk gospel show people how to use it if they've got questions all that sort of stuff and then the other part of that is really just showing people what we can do and advocating for the making decisions based on the data we have it in data you know I have it in spunk let's look at that to make the decision so that's really what that commander Awards kind of all about so if you're doing the doctor stuff you're a bit of a trailblazer so we were only a few years into this container initiative I was walking the show floor I even saw some companies looking at like the serverless technology you know what what led you to kind of put these pieces together and you know it tell us a little bit about kind of the community that you lean on to learn these things yes so the the technology trend around containers was very strong and very fast like with Amazon's especially like that when they came out with their ECS orchestration it was really fast and very strong and really the the technology trend kind of led me into it and then the developers being like we're gonna use docker we're gonna have to figure you're gonna have to figure out how to Splunk it so really from the very beginning I've gone through each and every sort of possible way to get data out of a dr. container in this Splunk and part of that is you know networking with the Splunk folks pretty good relationship with the with the fella that wrote the logging driver that went into the dock or open source project and like looked at the code reviews and all that and then it's really just trying it out trying things out and eventually kind of got to the sweet spot now where I've got the developers are all using local docker compose and that's configured a certain way then when we run in Amazon it's using Amazon ECS where I've also been working on kubernetes for a while and the way that you configure your docker in each of those environments is totally different the code running in the is exactly the same so we've realized that vision but the runtime environment is totally different so kind of where we're at now the config may be totally different on the logging drivers but in the end when you load up Splunk and you look at it and Splunk it's exactly the same whether it's your local laptop and amazon in production staving staging or whatever and I think my kind of favorite part in terms of like the Splunk commander award and getting folks using Splunk is that the way that I have it set it up set up now there is literally no local log file for the developers on their laptop it just doesn't exist it all goes out to Splunk so you can do a lot with grep and text pad and stuff on your local local laptop and I get that but now that they're in Splunk and it's just it's been a great way to get folks on board with what its gonna look like in production I know what it looks like in dev so I can make sure that my logs are good I'm logging enough and not too much and all that stuff so that's really where docker is really software is the same now we've got the logging the same the tools are all the same but then the runtime bits those are a little bit different and that's abstracted away hopefully Jim what does a DevOps guy want from a vendor you got a lot of open source stuff that you're working with you got a lot of different tooling what do you look for in a vendor what's what's the thumbs-up and positives and what what stuff really kind of ticked you off well so you know we're we're a key trusted vendor for a lot of healthcare organizations so I can kind of talk about how I we prison if a customer or a user comes up comes to us with a problem doesn't matter what it is it's our problem and we go through exhaustive lengths to identify where the problem actually is and so that may be in our code that maybe in another vendors code some third party some open source thing doesn't matter we're after the evidence we're after the facts we don't care if it's not in our code we're gonna help our customer be successful and that's what we would want from any vendor right so if we contact them with a support case we've got a problem we don't want any of this uh looks like a firewall problem or something like get to the data get to the facts and if you can prove if the vendor can prove that the problem is somewhere else great but we want a reproducible test case we want this whole finger-pointing thing is like it's horrible inside of an organization in terms of like running operational systems but then when you've got like as your Google Cloud Amazon Cloud Salesforce service now all these things all working together like you can't people just going to own the problem basically and that's what that's what we do right so if the customer comes to us with an issue it's our problem and then we go from there and figure it out and that's really what any vendor that we work with especially like a production operational sort of system that's really what we look for so you look for collaboration and focus on solving the problem not not the finger-pointing you know a virtual single throat to choke if you will yeah exactly hm well thanks very much for joining us on the cube is great to have you yeah thank you thank you very much appreciate I keep right - everybody stew and I'll be back hashtag Splunk conf 18 you're watching the cube right back [Music]