>> Live, from Las Vegas, it's theCUBE, covering AWS re:Invent 2018, brought to you by Amazon Web Services, Intel, and their ecosystem partners. >> Welcome back to AWS re:Invent, along with Justin Warren, I'm John Walls, we are live here in Las Vegas. Day two of our three days of coverage of this event, seventh time we've been here and, as we've been saying all along, this show is getting bigger and better than ever. About 40,000 attendees this year. Joined now by Abbas Haider Ali, CTO of xMatters, and Chris Crocco, who is the lead solutions architect at Viasat. Gentlemen, thanks for being with us, good to see you. >> Thanks for having me. >> Thanks for having us on. >> All right, tell us a little bit about your respective endeavors and then why the two of you are here together, and Abbas I'll let you lead off. >> Sure, I'm CTO at xMatters, as you described, and our company is basically a digital service availability platform which, outside the marketing speak, and from a technical perspective, means, when bad things happen with technology, and all technology's great but, inevitably, things go wrong-- >> Bad things happen. >> Bad things happen and we're in the business to helping companies get those things fixed as quickly as possible, ideally before they become business-impacting. And basically, I asked Chris here to join me because you can have technology but you need someone to put it into practice and Chris has done a great job of bringing it in to real world at Viasat. >> Good transition, thanks Abbas. My role at Viasat, Viasat's a satellite-based technology and communications company, and my role is to help administer and deploy some of our automations for orchestration monitoring performance and incident management. A lot of that has to do, as it relates to xMatters, with notifying people when they eventually have to go hands-on-keyboard and minimizing the amount of administrative burden that they have so they can just focus on fixing a problem. >> You mentioned before that everyone who was traveling here on an airplane, if they were using the wifi, they're probably running over your service? >> Right, yeah, so one of our-- >> I am astounded that that even works at all, speaking of technology breaking all the time. Maybe explain to us a little bit about how xMatters helps you keep that thing actually functioning. >> Yeah, that's a great question. One of the things that we monitor very, very tightly is our customer experience, both on aircraft, and residential broadband, and so when we're starting to see things where those planes are passing through beams and maybe not handing off the internet connectivity well, if we're seeing people trying to get on the internet and they're either having a slow time or just not getting on at all, one of the things that we want to do is get that to the right people quickly. So, one of the things that we do is we have our customer care elements of commercial mobility in xMatters so that they can report that to the engineering level for that same area of business. When they do that it's opening up a sales force case, it's notifying a Hipchat room, it's getting hold of the on-call resource, and it's also administrating all of that stuff back to the originator of the problem, so that they can keep them informed of "this is what the engineer found, "this is how long it's going to take to get fixed, "this is what you need to tell the customers." So it's enabling a lot of communication while reducing some of the traditional operational elements that go along with incident management. >> Yeah, it's something that we've been hearing quite a bit this week here at AWS, is the importance of that operations side of things. It feels like the whole industry has moved from this being a new technology that we should start doing brand new things with, and it's matured a little bit, where we're actually relying on this stuff to run real multi-billion dollar businesses and operations starts to become really, really important so, as you said, when things break, we want to fix them as fast as possible, so that customers can keep using our services. >> Right, and kind of in the path, when you look at all the companies that are here, they're building fantastic new products, builders are a big part of this event, it's all about building their services and you hear a lot about automation and tool change and the CI/CD pipeline. Well, the CI/CD pipeline really ends at delivery. And that's kind of where our product picks up. So it's in the operations and support realm of it is, once it's out there, things inevitably will go wrong and a lot of the companies you see here are all about detecting that very, very quickly. You'll hear conversations about one-second resolution in detect issues, and those things have to be handled. And really, one of the things that we're seeing a big trend in is going through and saying, "How do we remove the manual process, "and administrative overhead, and the toil "in actually operating these services, "when, inevitably, something goes wrong." And it starts off small and can grow very quickly, so a lot of people use our product, to essentially tie those alerting systems directly into xMatters, it goes out, gathers a lot of the information that people would typically do by hand, the manual effort, delivers it to the right on-call person and arms them with the action and move them through. And really, that cycle of steps, if you think of it as every individual team and service has a series of flows that they go through when things go wrong. It's about taking those steps and putting them all together in the right order and swapping them out as you need to as your service matures and grows. And as your innovation is successful and as you grow in scope, those steps may change, but the flows across the teams remain remarkably-- >> Is there-- >> The same. >> You talk about flows, different avenues, different opportunities, or problems, is there one that tends to stand out amongst the crowd as "That's our biggest headache," whether, for Chris's business, or just, in general, for any of your clients, is there one that leaves you scratching your head? >> If we go around and just interview all the various enterprises, who are consumers and builders of a service and we ask them, saying, "Hey, what's the single biggest thing that's kind of a pain when things go wrong?" One of the biggest problems that we see is that a lot of these organizations have built kind of a distributed operation model. And one of the biggest problems we see is, if you think of it as, you've got a whole series of things, a series of, kind of spokes, and one thing goes wrong, other people are consumers of it, and other people are impacted. All get engaged, saying my thing is also sending me a signal saying "My work has gone sideways," but it's very difficult to figure out where the actual responsibility lies and how do you engage just the people who could actually fix the issue and then let everyone else who is impacted by it be informed, but told to stand down, so they don't waste their cycles on resolving that. And that's a very complicated problem that there is no magical solution for so if anyone's listening and looking for "Okay, "that's what I've got, give me an answer," I don't have a solution for you (laughing) but I can tell you that a lot of these sorts of operational tasks we're putting in place are designed to minimize the effort of figuring out what that is and really speeding up that information cycle so you waste as little time as possible. >> Does that sound familiar, Chris? >> Very familiar, yeah. Viasat's company motto is "Always a better way" and so one of the things we do with xMatters and other tools in our incident response chain is take what we learn when we do have an incident, when we do have a problem, and find a better way of approaching that. It allows us to refine our integrations into xMatters. It allows us to communicate more effectively to the right people. It allows us to really kind of harness our DevOps model and that company credo to our advantage and constantly perform better for our customers. >> We were talking before we went live here, this is dealing with issues at the scale of space, so these sort of problems, and it's a theme that we've been hearing over the last couple of days, that the amount of complexity on these kinds of systems, and something at the scale of a space-based platform. This is something which isn't really tractable for the human mind to deal with unaided, so we really do need tools like xMatters to actually cope with this. But what has putting in something like xMatters done for the business of Viasat? What does that actually change, that you're now able to do that you weren't able to do before? >> Again, xMatters enables a lot of opportunity for our DevOps teams to constantly improve. One of the things that I personally like about xMatters a lot is it's not a centralized tool. A lot of tools in this space are intended for you to be constantly looking at a dashboard or have an incident captain that's always, their life is that tool. >> A single glass of pane. >> Right, but all of these teams have their own single pane of glass that they consume, so we can plug in xMatters where it's appropriate and allow those tool chains and those automation flows to include xMatters but not have it be the end all, be all of their process, so it helps them improve on all of the other parts of incident management and monitoring and xMatters is just there to facilitate those transactions and those workflows. So, a lot of value there, a lot of learning opportunities and a lot of enablement for all of our DevOps teams. >> So you can improve the way that you're doing things without having to rip out everything else and replace it with one new tool. >> Exactly, one of the things that you don't want to do in any organization is throw the baby out with the bathwater, so to speak. There are tools that can be refined and we see a lot of this in the trend toward micro-services, right? Instead of having vendor lock-in, this huge one-stop shop for everything, you can pull and replace all of the smaller pieces in that chain without affecting your availability or your ability to respond to an event. >> And one really interesting thing about these distributed models is you still have places where information needs to wind up, so if I'm working on a particular part of my application and I've got a customer service team that uses Salesforce as their system of choice, I have to get information to Salesforce so they can consume it. It's not okay for me to hoard information, I actually want to make sure that I'm minimizing the friction and moving information along to where it needs to wind up, along that process. If I am a developer, my kind of world view of my tasks are Jira, I want to make sure the information winds up there. If I'm in a service management team and I use something like ServiceNow, kind of track information there, I have to make information wind up there. We collaborate in Slack, I have to make sure that it's available within that world as well. So the key thing that we're really focused about is every team picks their own flows, they pick their own tools, but the steps along the way are very similar. Something goes wrong, you pull in the information, you need help, you need a collaboration step, and you need a basic information delivery stage to put information back in the right places because after it's done, to Chris's point, if you just solved the problem very effectively and learned nothing, you've done a bad job. We have to be clear about that, right? Learning and improvement is a key part of a successful DevOps transition, and when you're running things at the scale we're talking about at re:Invent, you have to learn. And a key part is making sure information winds up in the right places so you're able to do that. >> Getting them halfway happy won't cut it, right? >> Right, I would fully expect that Chris and other customers in Viasat's position would be like, "Yeah, that's great, we did it great this time, "but when it happens again, we would have learned nothing." >> What do we do next? >> Right, exactly. >> Right. >> Gentlemen, thank you for the time. We appreciate you sharing your story and wish you success. >> Thanks very much for having us on. >> For the rest of this week, enjoy the show. >> Thank you very much. >> Off to a great start, that's for sure. >> Thank you. >> Back with more from AWS re:Invent, with Justin Warren, I'm John Walls, and you're watching theCUBE. (upbeat music)

>> Live from Orlando, Florida, it's theCUBE, covering .conf2018! Brought to you by Splunk. (techno music) >> Welcome back to Orlando, everybody. We're here with theCUBE covering Splunk.conf2018. I'm Dave Vellante with my co-host, Stu Miniman. Chris Crocco is here, he's the Lead Solutions Engineer at ViaSat. Great to see you, thanks for coming on theCUBE. >> Well, thanks for having me. I appreciate it. >> You're very welcome. Let's start with ViaSat. Tell us what you guys do and what your role is all about. >> So ViaSat is a global communications and technology company primarily focused on satellite-based technologies, anything from government services to commercial aviation and residential service. >> And what does a Lead Solutions Engineer do? >> My primary role is to help us kind of transition from a traditional operations state into more of a DevOps environment including monitoring, alerting, orchestration and remediation. >> Oh, we love this conversation, don't we? Okay. The basic question is, and I know it's hard, but it's subjective, it's kind of if you think about the majority of your organization in the context of DevOps, on a scale of one to five, five being nirvana, so let's assume you're not at five 'cause it never ends, right? You're constantly evolving. Where would you say you are? Are you just getting started? Are you more like a four, 4 1/2, what do you think? >> That's a good question. I would say we're probably three on our way to four. We've had a lot of growing pains, we've had a lot of learning opportunities. The processes of DevOps are getting pretty well-entrenched and right now, we're working on making sure that the culture sticks with the DevOps. >> That's critical, right? >> I mean, that's really where the rubber meets the road is that organizational and political. Without getting into the dirt of it, give us what it looked like before and where you are today. >> Sure. Prior to our shift to DevOps, which was mainly motivated by our latest spacecraft, ViaSat-2, we had a very traditional operational model where we had everything funneled through a Network Operations Center, we had a Technical Operations Team, and if they weren't able to triage and remediate issues, they kicked it over the fence to engineers and developers who would then throw something back. There wasn't a lot of communication between the two organizations, so when we did find recurring problems, recurring issues in our network and in our environment, it took a long time to get those resolved and we had to have a large volume of staff there just to kind of put out the fires. With the transition of DevOps, one of the things that we've been focusing on is making sure that our development teams, our engineering teams understand the customer experience and how it's impacted by what they do, and de-centralizing that operation structure so all of the triage work goes to the people who actually work on those services. So it's a pretty big paradigm shift but it's also helping us solve customer problems faster and get better education about what the customer experience is to the people who actually make it better. >> And roughly, what was the timeframe that it took to go from that really waterfall model to the structure that you have today? >> We've been going for about two or three years now in this transition. Like I said, the first year or so was kind of bumpy and we've really kind of ramped up over the past year in terms of the amount of teams that are practicing DevOps, the amount of teams that are in an agile and scrum model. So overall, two to three years to get to where we are today. >> So the problem with the traditional model is you have time to deployment is slower, that means time, the value is slower, a lot of re-work. Here, you take it. No, you take it. Hey, it worked when I gave it to you, a lot of back and forth, and not a lot of communication creates frustration, not a lot of collaboration and teamwork, then you're working through that now. How large is the team? >> My team is five people. We have 4,500 people roughly at ViaSat as a whole. I believe roughly 2,000 of them are in an engineering or technical role. >> Okay, but in the previous model, you had developers and you had operations folks, is that right? And your five are sort of split over those or was it a much, much larger corpus of folks? >> It was a very large distribution of people. It was very engineering and developer-centric. We still had a Core Operations Team of 60 to 100 people based in our Denver office. We're keeping our headcount relatively the same with respect to our operations and we're growing a lot in terms of those DevOps teams. So as those teams continue to grow, we're adding more operational resources to them and kind of inserting a lot of that knowledge into other parts of the organization. >> You're doing a lot more with the same. Are you coming from the ops side or the dev side? >> I come from the ops side. I actually started my career with ViaSat in our knock in Denver. From there, I transitioned into a ops analyst role and then we created the Solutions Engineering Team and I took the lead on that. >> Chris, can you tell us how Splunk plays into your DevOps? Did you start using it in the knock and kind of go from there? >> We did, actually. Splunk started out as just a tool for us to see how many modems were offline in the knock. It was up on the video wall and we would see spikes and know that there was a problem. And as we've made this transition at DevOps, a lot of teams that were using other solutions, other open-source and home-grown solutions were kind of organically pivoting to Splunk because it was a lot easier for them to use for alerting dashboards, deep-data analysis, a lot of the things they needed to do their job effectively. So as we've grown as a company, as we've grown in this organizational model, Splunk has kind of grown along with that in terms of use case. >> That growth is predominately in IT operations and security, correct? >> Well, it's actually pretty interesting. It's kind of all over the board in our organization. It started in IT operations and security, but we have people in our marketing department using it to make sales and campaign decisions. We have executive leadership looking at it to see the performance of our spacecraft, we have exploratory research being done with it in terms of what's effective and what's not for our new spacecraft that will be coming out, the ViaSat-3 Constellation. So it's really all over the board in our organization. >> That's interesting, Stu, you're not the first customer who's told us that no, it's not just confined to IT, it's actually seeping through the organization. Despite the fact that we heard a bunch of announcements today, I don't know if you saw the keynotes, making it simpler for lines of business folks to actually utilize Splunk, so given that a lot of your teams in the business are actually using it already, what do you think these announcements will do for them? Maybe you haven't had time to evaluate it, but essentially, it's making it easier for business people, you know, simplifying it. >> Yeah, you know, all of the announcements in the keynotes over the past two days have been really, really exciting. Everything that I was hoping for got checked off the list. So I think one of the big things that it's going to allow us to do is get our customer-facing teams and our customer care organizations more involved with the tool. And getting them the information that they need to better serve customers that are calling in, and potentially even prevent the situations that customers have to call in for in the first place. So giving them a lot of account information quickly, giving them the ability to access information that is PCI and PII-compliant but still allowing them to get the data they need to service an individual customer, all of those things I think are really going to be impacted by the announcements in this conf. >> So you were the keynote yesterday. >> I was! >> Were you shaking the phone? >> I was, yeah. >> Which group were you, were you orange? >> We were orange group, yeah. >> We were orange, too! But we were sitting in the media section and all the media guys were sitting on their hands but we had a lot of devs and ops guys shaking with us. It's like when you do the wave at Fenway Park when it gets behind home plate, everybody just kind of sits down, but we were plugging hard. Alright, Chris, what else has excited you about .conf2018? School stuff that you've seen, some innovations, things you've learned. >> Well, I'm really excited about the app for infrastructure. That's something that we've been trying to get for ITSI for a long time now in terms of NED-level monitoring and NED-level thresholding. I think that's going to complement our business really, really well. The advancements that they're doing with the metrics store, specifically with things like Syslog are really, really exciting. I think that that's going to allow us to accelerate our data and make it more performant. The S3 compliant storage is absolutely fantastic and it comes in black now and that's really, really fantastic. >> Oh right! The dark mode! >> Dark mode, yup. >> You mentioned the ITSI. Have you used the VictorOps pieces before or is that something you're looking to do? >> We haven't looked at VictorOps as of yet. We're an xMatters customer right now so we've been using their integration that they built out and it's on Splunk base. But VictorOps, it'll be interesting to see how that organization changes now that it's part of the Splunk. >> So dark mode actually, it's one of those things that it really got such a loud ovation. It was funny, I was actually talking to a couple Splunkers that are like, "We want that dark mode t-shirt." Which I think you have to be a user and you need to sign up for some research thing that they're doing, and they're giving out the black shirt that has like gray text on it. >> Awesome! >> Why does that resonate with you, the dark mode? >> Well, it was actually what they talked about in the keynote. If you have it up on a video wall, which we have in various parts of our company, or if you're sitting in a dark office, something like that, looking at a really white screen for a long period of time, it's not easy on your eyes, it's hard to look at for a long period of time. And generally speaking, a lot of our presentation layers go towards that visual format. So I think this is going to allow us to make it much more appealing to the people who are putting this up on screens in front of people. >> Your responsibility extends out into the field, I presume. The data that's in the field, is that true? >> It does. >> Okay, so I'm interested in your reaction to the industrial IoT announcements, how you see or if you see your organization taking advantage of that. >> Well, we're a very vertically integrated company so we actually manufacture a lot of the devices that we use and that we provide to our customers. I think a lot of our manufacturing capabilities would really benefit from that. Anything from building antennas for ground segment that actually talked to the spacecraft. It's the modems that we put in people's houses, that entire fabrication process I think would benefit a lot. I really loved the AR presentation that they did where they were actually showing the overlay of metrics on a manufacturing line. I think that's something that would be fantastic for us, particularly for sending somebody to an antenna or a ground station to replace a piece of equipment. We can overlay those metrics, we can overlay all of that, we can use the industrial analytics piece of that to actually show which piece of hardware is most affected and how best to replace that. So a lot of opportunities there for our company. >> So I wonder if you could help us understand what's, from your perspective, on Splunk's to-do list. We're going to have Doug Merritt on a little later. If you had Doug right here and he said, Chris, what can we do to make your life better? What would you tell him? >> You know, I think a couple of the things that would make it better, and it looks like they're heading this direction, is streaming in and streaming out. You know, streaming in is of course important, that's where a lot of your data lives, but you also have to be able to send that out to Kafka, to Kinesis, to other places, so other people can consume the output of what Splunk is doing. So I think that would be a really, really important thing for us to socialize the benefit of Splunk. And then vertically integrating the incident management chain, it looks like something that's on their roadmap and I'd be interested to see what their roadmap looks like in terms of pulling in Phantom, pulling in VictorOps, pulling in some of these other technologies that are now in the Splunk umbrella to really make that end-to-end process of detecting, directing and remediating issues a lot more efficient. >> Okay, and do you see at some point that the machine will actually do, the machine intelligence will do a lot of that remediation? >> I think so. >> Do you see the human still heavily involved? >> Well, I think one of the important things is for a lot of these remediation things, we shouldn't have a human involved, right? Particularly things that are well-known issues. Human beings are expensive and human beings are important, and there are a lot more important things that they can be doing with their time than putting out fires. So if we can have machines doing that for them, it frees them up to do a lot more cool stuff. >> You're right. Alright, Chris, well listen, thanks very much for coming on theCUBE. It was great to have you. >> Yeah! Appreciate it very much. >> Thanks for your insights. Alright, keep it right there, everybody. Stu and I will be back with our next guest. You're watching theCUBE from Orlando Splunk.conf2018. Be right back. (techno music)