[upbeat electronic music] >> Hi everyone, welcome back to theCUBE's coverage of AnsibleFest 2021. I'm John Furrier, your host of theCUBE. We're here with Eric Pennington, Director of Solutions Engineering, and Mike Todaro, Senior Epic Cache Consultant at Sapphire Health. Gentlemen, thank you for coming on theCUBE and chatting about the wave of Cloud, cloud-native, Sapphire Health and Ansible. Thanks for coming on. >> Thanks for having us. >> Thank you. >> So, let's get started. Can you guys just briefly describe Sapphire Health and what you guys are doing there. The consulting services, the trends that you're seeing. Just take a step, a minute to describe the environment at Sapphire Health and what you guys are doing. >> For sure, yeah. So, Sapphire Health was a consultancy that was founded by the CEO back in 2016, Austin Park, who also serves as a CTO for some healthcare organizations, because he was having difficulty finding an organization that really specialized in Epic infrastructure. So you might be familiar with some of the large players in Epic consultancies, but they are typically focused more on the application side, so configuring like the ambulatory clinical system or something like that. And there really wasn't a solution that he could find in the market for an organization that was focused on Epic infrastructure and some of the more technical components of managing an Epic technical ecosystem. So, Austin founded a team. Mike was one of the early folks to join. I joined a little bit later. But he put a team together to, again, really focus on the technical components of an Epic implementation. And since then, we've been providing managed services for Epic infrastructure for a number of organizations. We've been focusing on platform migrations from, for example, AIX to REL for Epic organizations, and we've been focusing on some growth areas as well in the Cloud. Epic systems is now able to be hosted on the public Cloud, that's a relatively recent occurrence. So, we're working with some organizations in that space as well. Mike, anything you'd add there? >> No, I think that pretty much covers it. We've spent a large fraction of our effort making sure that we're engineering solutions for these clients that move them in the directions towards Cloud readiness, towards containerization, automation, and those sorts of things. I think Eric's description's spot on. >> So, you guys must be busy. I mean, I can only imagine the action happening right now as people realized, with the pandemic specifically, two areas that we've reported aggressive growth on was public sector and healthcare. Both were under massive strains of pressure to get faster. (chuckles) Can you guys just weigh in real quickly on what you guys are seeing and how that's impacted your consulting services, but also the customer. What's going on in their minds? >> Absolutely, we had some customers very early on in the beginning of the pandemic where we were given the cadence of updates coming from Epic, the needs for growth for those customers where both in ICU surge capability as well as just general admittance. There was a flurry of hardware purchasing, provisioning, set up. An increased cadence around patching for various pieces of the Epic environment including Epic code directly. All of those things. The tempo of all of that increased once the pandemic began, and we spent a significant fraction of time trying to find better ways, faster ways to engineer what we were already doing for clients, simply so that we could continue to keep up with the surge in demand without requiring an additional surge in investment in people, where it wasn't necessary. Obviously, some growth was necessary, but we wanted to help our clients get the most out of what they already had so that they could spend that money where it was needed to help patients. >> Yeah, awesome, great stuff. So, we're here at AnsibleFest getting into the action. It's all about automation. So I have to ask you guys, what led you to start exploring automation solutions at Sapphire Health? >> Yeah, so there's quite a few reasons. I would say the most critical is that we've been providing managed services to organizations around infrastructure management for some time. And as you can imagine, infrastructure management has some repetitive tasks, and I'm quoting my colleague, Mike, here, but a good administrator is a lazy administrator. And what we mean when we say that is, if there's a repetitive task that's being performed over and over again, if there's an opportunity to automate it, that's going to save us time. But more importantly, that's going to... Paul, these lights here. Let me move around a little bit, should come back, there we go. But it's going to provide an opportunity for us to focus on more value-add services for the client. It's going to reduce costs for the client in terms of the services that we're providing. And I think most importantly, it's removing the possibility for human error or the possibility for error overall. So it's a natural evolution of us observing the time that we're spending with our client partners, and again, it really provides a lot of value to Sapphire as an organization and our customer partners as well. >> Mike, you want to weigh in on this automation trend. How do you see it evolving? I mean, obviously sounds good when you want to automate things that you do repetitive tasks, but is there more going on that you see in automation that goes beyond just, okay, if you do it three times-automated kind of vibe. >> Sure. Automating repetitive tasks is the kiddie end of the pool. That's how we get... That's how we sell the idea to people who just don't get the concept yet. But there are workflows that really aren't feasible outside of automation. We tend to think of automation, in some cases in this sort of limited way, but automation is really... What we really are targeting with automation is more about workflow. It's less about individual tasks, and it's more about an idea of workflow or a business requirement from its origin all the way through its implementation. So, I've got just the simplest case that jumps immediately to mind, is I have a new hire, I've got to provision them an account. I need to provision it across multiple systems. I've got to do it in our single sign on. They need home directories. They might need access. They need building accesses we need to generate. You got to generate badges for these people. And these are all workflows that are normally disparate. You know, you have to take your sheet to this guy, take your sheet to this guy, here's my new hire form. Really, what you really want is, we got a new hire, everything's checked out, put it in this basket here and let the automation move it through all of these systems all the way across. And that's the sort of thing, like I said, that's a very limited, very simple idea, but that's the kind of thing we really want. We want to get in the door with automation with simple things and then we want to teach... We want clients and ourselves to be challenged, to be creative, to find new ways to apply it that aren't immediately obvious. >> Yeah, I was smiling because I love the example of the kiddie end of the pool because automation is going mainstream, and it used to be kind of, you know, for the geeks who were doing the hardcore stuff who got the whole big picture. Now you're seeing with AI automation moving in and with Cloud, a lot more automation happening. So, I can almost see in my mind mental image of people wearing bubbles in the pool, kind of like going in the deep end, get back over here. Stay in your lane. Yeah, but this is the trend, and I want to get into this because you guys are involved in this Epic migration that's been talked about. So for the folks that aren't in, say the health care space, put a little context around Epic and then I want to get into this whole migration discussion. I think that kind of points to some real value propositions. So, what is Epic for the folks outside healthcare? >> Sure, so Epic is one of the leading EHRs or electronic health records software in the world. It is by far the most deployed in the United States. What's involved in building an Epic, or performing an Epic migration. Epic is hundreds of systems. When you think about Epic as an umbrella concept, it is servers and end-user workstations and all of these things. When we talk about platform migration, what we're usually talking about is the transactional database. They call it the ODB or whichever term I think you feel applies best. When we perform all those migrations, we're usually talking about... When we perform one of those migrations, we're usually talking about an AIX to Red Hat migration, although you can just do hardware to hardware. Involved in that is a number of things. You're building new VMs. You're setting up patch cycles, setting up the patching server. Installing the various administration scripts that Epic provides. Installing the software that runs the DB, which at the moment is either InterSystems Cache or Iris. There's the provisioning of the local security users. There's the configuration of the OS. If you're moving from AIX to Red Hat, you're talking generally about a bit endians conversions, so, big endian to little endian, there's a tool for that. There's a lot of these little stats. And the thing is, is that, they're all very, very well defined and very similar, and so, they look identical in many of these cases from one implementation of Epic to the next. And that's not true for the entire Epic stack necessarily, but at the ODB level, this stuff is all very similar, and this is a very right place to automate. This screams automate, and we do this because, I mean, who wants to make mistakes. If you write and build your script and debug it, the script runs, it doesn't make mistakes. I make mistakes, the script doesn't. So, we do that, and we end up spending less time on these repetitive, unnecessary tasks. We guarantee the correctness of them, or we do a better job of guaranteeing the correctness of them, and all of that ends up saving money in the long run. >> That's awesome, and thanks for the context. I was going to get there on the automation piece. It really sets the table for the automation. Real quick clarification. How much or what kind of software work is involved in a migration? >> Oh, so there's the installation of... You have from the installation of the OS and the configuration of the OS, the building in the patch server, the implementation, testing, and patch cycling. There's those data conversions I talked about. There's environment refreshes where we copy an existing environment on a regular basis to another environment for things like testing, for troubleshooting purposes or for other reasons. There's more than one database for Epic. There's one big production database. You have training databases, and you have playground databases for people to work in so they can learn to use the system better, and then there are, I mean, there's a galaxy. >> Oh man, so it's a huge system. Okay, so I got to ask the security question. >> Sure. >> Is security element as important when selecting automation or how has that factored in? I mean, right now that's super important, obviously, records are key, but honestly, where does that fit into the automation piece of security? >> Yeah, I think that's a very important question, and as you alluded to, security is incredibly important. It's very important in healthcare in particular. And in fact, with healthcare, there's a lot of regulatory requirements. There's a lot of requirements that individual healthcare institutions have that we as a partner to that institution need to follow. So, as we were evaluating automation vendors and automation solutions, a highly secure system was not a nice to have or like a value add, it was something that was absolutely critical and paramount to being able to successfully automate any of the things that we're doing. So I'll turn it over to Mike to talk about some of the specifics, but as we evaluated Ansible, we saw that it really supported robust security. So, Mike, can you comment a little bit more on that? >> Sure. There's a number of ways that we use Ansible to help improve the security posture for clients. One of the ways is Ansible playbooks are written to be runnable against the server and nothing will change unless something is set incorrectly. And this lets us assure that the configuration is where we expect it to be so we don't get drift on these servers. Now, remember I said an Epic environment is a lot of servers. If one or two of these... >> John: Mike, if you don't mind, I need to interrupt. What is, when you say drift, what are you referring to? >> So when I say drift, what I mean is, if there's a bunch of different servers and I as an administrator have to work on one or two of these servers just for little things during the day, I might make a change on one of these servers advertently or inadvertently, and then that server's configuration is now slightly out of phase with the other servers, which could be benign, but it could also be a security hole. Having Ansible able to run nightly and continue to adjust these servers back to the expected baseline, and in the case of things like tower, be able to report that these things were out of position. Let us know, hey, it lets us reduce the attack surface, first of all. It lets us multiply it, like a force multiply our attention across this farm of servers, and it gives us that sort of clarity that we know we're doing what we have to do to make sure these servers continue to be safe. >> That's an awesome service. That right there is, I mean, just going in manually trying to figure all this stuff out, it's just a nightmare. I mean, what a great relief that is. I mean, just the alternative is what, you know, more pain and suffering human wise, that's the labor, and then risk on attack because people go to bed. >> I'm a patient. The thing is, on a personal note, I'm a patient too, all of us are. We all have doctors. We have to go to the hospital for things occasionally. And if we fail when we perform these security audits, if we fail when we perform these security checks, patient data can get lost. It can get sent to people who shouldn't have it. And I'm a patient, I have no desire for my medical information to be available anywhere but in the hands of my doctor or myself. And that's the thought I try to stay with when I'm working on these systems. I'm a patient. It's not that I'm doing this because... I mean, the knock-on effects of reducing liability for the customers cannot be ignored or overstated, and they're critical, but, ultimately, my eyesight is on the patient. >> Yeah and having that stability is huge. Okay, this brings up the whole automation thing as it becomes more mainstream for you guys, specifically, is critical. The system's there, you have to watch farms, all the action happening, it's a huge system. Complex automation is key. How are you guys continuing to push the automation envelope into the Sapphire Health's consulting practice? >> Well, as you mentioned, John, yeah, we're really taking a look at the entire technical infrastructure when we're working with our clients. And we are offering fully outsourced managed services for organizations, not just around the Epic infrastructure but things like networking devices, security and other third party systems. So with that, we're seeing a lot of these things that are going on, and we're always evaluating opportunities for automation. There's actually two areas in particular that we're seeing gain a lot of momentum with our customers, and we're seeing a lot of opportunity for automation. The first is business continuity and disaster recovery, specifically within Epic. So, Epic has very stringent requirements for resiliency, as you can imagine. When the system goes down, a hospital can't really do what it needs to do from a billing standpoint, a clinical standpoint, so very robust disaster recovery and resiliency standards and solutions are very important. However, there's not a lot of automation that's available either from Epic or, as far as I know, other consultancies, so what we did is we built a script that provides failover automation. So some of the tasks that would be very manual in terms of failing over to your DR solution, we've automated that, and that again, removes a lot of the opportunity for human error, really speeds up the failover process. And so with the customers that we work with, that's something that we provide. Another big area that we're seeing is environment refreshes. So within Epic, there are different environments that are, basically, all their data is copied over on a recurring basis from the production environment, and the refreshes can have a lot of manual steps involved, so we found an opportunity and have implemented some automation around environment refreshes for some of our managed services clients. And as we continue to go throughout, you know, building our Cloud practice in some other areas, I'm very confident that we're going to see, you know, infrastructure is code more opportunities for automation around areas like that. >> I mean, you guys got to love the DevOps vibe going on now. Mike, I mean, you guys have seen the movie before in the old legacy going back to the mainframes, so you probably still run into a lot of older systems that still do a purpose. I mean, I have a lot of friends and clients that are working in the big banks, and they still have all the old school that does their job well, but containerization and Cloud kind of give life to these systems because now we're living in this system architecture called distributed computing again with the Cloud. It's the same game, different, different stuff though. >> Absolutely. Years ago, almost every Epic client was running on AIX, and maybe not mainframe but more mini computer. The migration path for almost all of the clients has been to move from those AIX mini computers down to VMs running Red Hat, or running Linux, and the natural evolution of that path is to move at least disaster recovery data centers into the Cloud, and then for some clients, the economics say the whole data center to the Cloud. So, absolutely that path is, it's well forged, it's there. I suspect that we'll see a lot more of clients, even larger hospitals, beginning to move down that road in the near future. >> And for the folks watching who may not have the scar tissue that we have, AIX was IBM's old Unix, a kind of mid-range mini computer. It was kind of client server, it was client server going now again being modernized. So obviously Red Hat is now part of IBM, but it speaks not just to IBM, this is about Ansible, right. So this is like, there is action happening here, so this is a case study of pretty much all migrations. It's not just the fact that it's AIX to Red Hat, it's system to the new thing that has benefits. >> Absolutely. >> What's your take, Mike, on that that kind of paradigm, because a lot of people going through similar situations just change AIX to something else. You have a lot of this migration re-platforming going on with the opportunity to kind of tweak it and add stuff to it. What's your advice and what's your reaction to this big trend? >> My advice for this trend, honestly, my advice is when you're planning these migrations, you know they're coming. Even if you're not in the cycle yet, you know it's coming. My advice is start brainstorming your implementation of the automation now. Get your automation into the system as you platform into your new platform, because it is far easier to build that entire platform with automation as a critical component than it is to bolt it on later, and you will get much more out of your investment and time and effort if you've integrated it from the very beginning. I would say anyone that was looking to perform a platform migration now and hadn't already begun serious consideration of running automation or had no plans for an automation, was setting themselves up for a very long and very difficult road to hell, and I would advise against it at this point. >> Great, great insight, Mike and Eric. Thanks for coming on, appreciate your insight here. You guys want to give a quick plug for the company? What you guys are looking to do, hiring, any update you want to share because great, great content you guys just shared here. Thanks for doing that. Take a minute to put a plug for the company. >> Yeah, I think a quick plug here. Yeah, if you're a talented cache admin, there's not too many Mikes out there, so we're definitely looking for more Mikes. But more broadly, we're really looking to expand into the Cloud space. We're rapidly expanding our managed services opportunities, and what we're seeing is a lot of organizations have like one ODB admin or one client systems ECSA admin. And what they run into is that person will leave, that person will retire, that person needs to get married and go on their honeymoon. It's kind of a problem, so we're working with a lot of organizations to not just fully outsource their environment but to provide a hybrid-managed service to provide overflow, to provide capabilities, to scale up with upgrades and projects like that. So, talk to us, we're pretty darn good at it, as you heard from Mike. We've got a couple of Mikes, again, we could use more, so if you are a Mike, please reach out. >> I think we virtualized him, we just virtualized Mike, you know, virtualization is a huge trend. >> If data writes Mike, we need to do that, yeah. >> Are you a body, are you the real Mike? >> (laughing) As far as I know, my wife would appreciate it if you guys would clone me a few times. >> You know, I've heard horror stories, Eric, around root passwords, like, who has the root password, oh, she left two years ago, kind of situations, this happens. I mean, this is not... it sounds like crazy but people leave. >> Yeah, I mean, nobody works anywhere forever, right? >> Don't be that company where you lose the root password, and never mind the ransomware action. Oh my God, must be brutal. Anyway, we can go another segment on that. Eric, thank you for coming on. Mike, thank you for your insight, really appreciate it, thanks for coming on. Appreciate it. >> Absolutely. >> Absolutely, it was our pleasure. >> Stay right here for continued coverage of AnsibleFest 2021. This is theCUBE, I'm John Furrier. Thanks for watching. (slow tempo electronic music)