>> Announcer: From theCUBE studios in Palo Alto and Boston, connecting with thought leaders all around the world, this is a CUBE Conversation. >> Hello and welcome to this special CUBE Conversation here in the Palo Alto studios, I'm John Furrier, host of theCUBE, for this Leading with Observability series with Under the Hood with Splunk Observability, I'm John Furrier with Arijit Mukherji with Splunk, he's a distinguished engineer, great to have you on. These are my favorite talks. Under the Hood means we're going to get all the details, what's powering observability, thanks for coming on. >> It's my pleasure, John, it's always nice to talk to you. >> Leading with Observability is the series, want to take a deep dive look across the spectrum of the product, the problems that it's solving, but Under the Hood is a challenge, because, people are really looking at coming out of COVID with a growth strategy, looking at cloud-native, Kubernetes, you're starting to see microservices really be a big part of that, in real deployments, in real scale. This has been a theme that's been growing, we've been covering it. But now, architectural decisions start to emerge. Could you share your thoughts on this, because this becomes a big conversation. Do you buy a tool here, how do you think it through, what's the approach? >> Exactly, John. So it's very exciting times in some sense, with observability right now. So as you mentioned and discussed a few times, there's a bunch of trends that are happening in the industry which is causing a renewed interest in observability, and also an appreciation of the importance of it, and observability now as a topic, it's like a huge umbrella topic, it covers many many different things like APM, your infrastructure monitoring, your logging, your real user monitoring, your digital experience management, and so on. So it's quite a set of things that all fall under observability, and so the challenge right now, as you mentioned, is how do we look at this holistically? Because, I think at this point, it is so many different parts to this edifice, to this building, that I think having a non-integrated strategy where you just maybe go buy or build individual pieces, I don't think that's going to get you very far, given the complexity of what we're dealing with. And frankly, that's one of the big challenges that we, as architects within Splunk, we are scratching our heads with, is how do we sort of build all of this in a more coherent fashion? >> You know, one of the things, Arijit, I want to get your thoughts on is because, I've been seeing this trend and, we've been talking about it on theCUBE a lot around systems thinking, and if you look at the distributed computing wave, from just go back 20 years and look at the history of how we got here, a lot of those similar concepts are happening again, with the cloud, but not as simple. You're seeing a lot more network, I won't say network management, but observability is essentially instrumentation of the traffic and looking at all the data, to make sure things like breaches and cybersecurity, and also making systems run effectively, but it's distributed computing at the end of it, so there's a lot of science that's been there, and now new science emerging around, how do you do this all? What's your thoughts on this, because this becomes a key part of the architectural choices that some companies have to make, if they want to be in position to take advantage of cloud-native growth, which is multifold benefits, and your product people talk about faster time to market and all that good stuff, but these technical decisions matter, can you explain? >> Yes, it absolutely does. I think the main thing that I would recommend that everybody do, is understand why observability, what do you want to get out of it? So it is not just a set of parts, as I mentioned earlier, but it brings direct product benefits, as we mentioned, like faster mean time to resolution, understanding what's going on in your environment, having maybe fewer outages at the same time, understanding your causes, so many different benefits. So the point is not that one has the ability to do maybe (indistinct) or ability to do infrastructure (indistinct), the main question is aspirationally, what are my goals that are aligned to what my business wants? So what do I want to achieve, do I want to innovate faster? In that case, how is observability going to help me? And this is sort of how you need to define your strategy in terms of what kind of tools you get and how they work together. And so, if you look at what we're doing at Splunk, you'll notice it's extremely exciting right now, there's a lot of acquisitions happening, a lot of products that we're building, and the question we're asking as architects is, suppose we want to use, that will help us achieve all of this, and at the same time be somewhat future-proofed. And I think any organization that's either investing in it, or building it, or buying it, they all would probably want to think along those lines. Like what are my foundational principles, what are the basic qualities I want to have out of this system? Because technologies and infrastructures will keep on changing, that's sort of the rule of nature right now. The question is how do we best address it in a more future-proofed system? At Splunk, we have come up with a few guiding principles, and I'm sure others will have done the same. >> You know, one of the dynamics I want to get your reaction to is kind of two perspectives, one is, the growth of more teams that are involved in the work, so whether it's from cyber to monitoring, there's more teams with tools out there that are working on the network. And then you have just the impact of the diversity of use cases, not so much data volume, 'cause that's been talked about, lot of, we're having a tsunami of data, that's clear. But different kinds of dynamics, whether it's real-time, bursting, and so when you have this kind of environment, you can have gaps. And solar winds have taught us anything, it's that you have to identify problems and resolve them, this comes up a lot in observability conversations, MTTI, mean time to identify, and then to resolve. These are concepts. If you don't see the data, you can't understand what's going on if you can't measure it. This is like huge. >> Yes, absolutely right, absolutely right. So what we really need now is, as you mentioned, we need an integrated tool set, right? What we mean by that, is the tools must be able to work together, the data must be able to be used across the board. So like by use case it should not be siloed or fragmented, that they should work as one system that users are able to learn, and then sort of be able to use effectively without context switching. Another concept that's quite important is, how flexible are you? Are you digging yourself into a fixed solution, or are you depending on open standards that will then let you change out implementations, or vendors, or what have you, (static crackles) down the line, relatively easily. So understanding how you're collecting the data, how good can open standards and open source you're using is important. But to your point about missing and gaps, I think full fidelity, like understanding every single transaction, if you can pull it off, is a fascinating superpower, because that's where you don't get the gaps, and if you are able to go back and track any bad transaction, any time, that is hugely liberating, right? Because without that, if you're going to do a lot of sampling, you're going to miss a huge percentage of the user interactions, that's probably a recipe for some kind of trouble down the line, as you mentioned. And actually, these are some of those principles that we are using to build the Splunk Observability Suite, is no sample or full fidelity is a core foundational principle, and for us, it's not just isolated to, let's say application performance management, where user gets your API and you're able to track what happened, we are actually taking this upstream, up to the user, so the user is taking actions on the browser, how do we capture and correlate what's happening on the browser, because (indistinct) as you know, there's a huge move towards single-page applications, where half of my logic that my users are using is actually running on the browser, right? And so understanding the whole thing end to end, without any gaps, without any sampling, is extremely powerful. And so yes, so those are some of the things that we're investing in, and I think, again, one should keep in mind, when they're considering observability. >> You know, we were talking the other day, and having a debate around technical debt, and how that applies to observability, and one of the things that you brought up earlier about tools, and tool sprawl, that causes problems, you have operational friction, and we've heard people say "Yeah, I've got too many tools," and just too much, to replatform or refactor, it's just too much pain in the butt for me to do that, so at some point they break, I take on too much technical debt. When is that point of no return, where someone feels the pain on tool sprawl? What are some of the signaling where it's like, "You better move now (indistinct) too late," 'cause this integrated platform, that's what seems to be the way people go, as you mentioned. But this tool sprawl is a big problem. >> It is, and I think it starts hitting you relatively early on, nowadays, if you ask my opinion. So, tool sprawl is I think, if you find yourself, I think using three or four different tools, which are all part of some critical workload together, that's a stink that there's something could be optimized. For example, let's say I'm observing whether my website works fine, and if my alerting tool is different from my data gathering, or whatever, the infrastructure monitoring metrics tool, which is different from my incident management tool, which is different from my logs tool, then if you put the hat on of an engineer, a poor engineer who's dealing with a crisis, the number of times they have to context switch and the amount of friction that adds to the process, the delay that it adds to the process is very very painful. So my thinking is that at some point, especially if we find that core critical workloads are being fragmented, and that's when sort of I'm adding a bunch of friction, it's probably not good for us to sort of make that sort of keep on going for a while, and it would be time to address that problem. And frankly, having these tools integrated, it actually brings a lot of benefit, which is far bigger than the sum of the parts, because think about it, if I'm looking at, say, an incident, and if I'm able to get a cross-tool data, all presented in one screen, one UI, that is hugely powerful because it gives me all the information that I need without having to, again, dig into five different tools, and allows me to make quicker, faster decisions. So I think this is almost an inevitable wave that everybody must and will adopt, and the question is, I think it's important to get on the good program early, because unless you sort of build a lot of practices within an organization, that becomes very very hard to change later, it is just going to be more costly down the line. >> So from an (indistinct) standpoint, under the hood, integrated platform, takes that tool sprawl problem away, helps there. You had open source technology so there's no lock-in, you mentioned full fidelity, not just sampling, full end to end tracing, which is critical, wants to avoid those gaps. And then the other are I want to get your thoughts on, that you didn't bring up yet, that people are talking about is, real time streaming of analytics. What role does that play, is that part of the architecture, what function does that do? >> Right, so to me, it's a question of, how quickly do I find a problem? If you think about it, we are moving to more and more software services, right? So everybody's a software service now, and we all talk to each other in different services. Now, any time you use a dependency, you want to know how available it is, what are my SLAs and SLOs and so on, and three nines is almost a given, that you must provide three nines or better. Ideally four nines of availability, because your overall system stability is going to be less than the one of any single part, and if you go to look at four nines, you have about four or five minutes of total downtime in one whole month. That's a hard thing to be able to control. And if your alerting is going to be in order of five or 10 minutes, there's no chance you're going to be able to promise the kind of high availability that you need to be able to do, and so the fundamental question is you need to understand problems quick, like fast, within seconds, ideally. Now streaming is one way to do it, but that really is the problem definition, how do I find the problems early enough so that I can give my automation or my engineers time to figure out what happened and take corrective action? Because if I can't even know that there's something amiss, then there's no chance I'm going to be able to sort of provide that availability that my solution needs. So in that context, real time is very important, it is much more important now, because we have all these software and service dependencies, than it maybe used to be in the past. And so that's kind of why, again, at Splunk, we invested in real time streaming analytics, with the idea again being, let the problem, how can we address this, how can we provide customers with quick, high level important alerts in seconds, and that sort of real time streaming is probably the best way to achieve that. And then, if I were to, sorry, go ahead. >> No, go on, finish. >> Yeah, I was going to say that it's one thing to get an alert, but the question then is, now what do I do with it? And there's obviously a lot of alert noise that's going out, and people are fatigued, and I have all these alerts, I have this complex environment, understanding what to do, which is sort of reducing the MTTR part of it, is also important, I think environments are so complex now, that without a little bit of help from the tool, you are not going to be able to be very effective, it's going to take you longer, and this is also another reason why integrated tools are better, because they can provide you hints, looking at all the data, not just one type, not just necessarily logs, or not just necessarily traces, but they have access to the whole data set, and they can give you far better hints, and that's again one of the foundational principles, because this is in the emergent field of AIOps, where the idea is that we want to bring the power of data science, the power of machine learning, and to aid the operator in figuring out where a problem might be, so that they can at least take corrective action faster, not necessarily fix it, but at least bypass the problem, or take some kind of corrective action, and that's a theme that sort of goes across our suite of tools is, the question we ask ourselves is, "In every situation, what information could I have provided them, what kind of hints could we have provided them, to short circuit their resolution process?" >> It's funny you mention suite of tools, you have an Observability Suite, which Splunk leads with, as part of the series, it's funny, suite of tools, it's kind of like, you kind of don't want to say it, but it is kind of what's being discussed, it's kind of a platform and tool working together, and I think the trend seems to be, it used to be in the old days, you were a platform player or a tool player, really kind of couldn't do both, but now with cloud-native, as it's distributed computing, with all this importance around observability, you got to start thinking, suite has platform features, could you react to that, and how would you talk about that, because what does it mean to be a platform? Platforms have benefits, tools have benefits, working together implies it's a combination, could you share your thoughts on that reaction to that? >> That's a very interesting question you asked, John, so this is actually, if you asked me how I look at the solution set that we have, I will explain it thus. We are a platform, we are a set of products and tools, and we are an enterprise solution. And let me explain what I mean by that, because I think all of these matter, to somebody or the other. As a platform, you're like "How good am I in dealing with data?" Like ingesting data, analyzing data, alerting you, so those are the core foundational features that everybody has, these are the database-centric aspects of it, right? And if you look at a lot of organizations who have mature practices, they are looking for a platform, maybe it scales better than what they have, or whatnot, and they're looking for a platform, they know what to do, build out on top of that, right? But at the same time, a platform is not a product, 99% of our users, they're not going to make database calls to fetch and query data, they want an end to end, like a thing that they can use to say, "Monitor my Kubernetes," "Monitor my Elasticsearch," "Monitor my," you know, whatever other solution I may have. So then we build a bunch of products that are built on top of the platform, which provide sort of the usability, so where, it's very easy to get on, send the data, have built-in content, dashboard (indistinct), what have you, so that my day to day work is fast, because I'm not a observability engineer, I'm a software engineer working on something, and I want to use observability, make it easy for me, right? So that's sort of the product aspect of it. But then if you look at organizations that a little bit scale up, just a product is also not good enough. Now we're looking at a observability solution that's deployed in an enterprise, and there are many many products, many many teams, many many users, and then how can one be effective there? And if you look at what's important at that level, it's not the database aspect or the platform aspect, it's about how well can I manage it, do I have visibility into what I am sending, what my bill is, can I control against incorrect usage, do I have permissions to sort of control who can mess with my (indistinct) and so on, and so there's a bunch of layer of what we call enterprise capabilities that are important in an organizational setting. So I think in order to build something that's successful in this space, we have to think at all these three levels, right? And all of these are important, because in the end, it's how much value am I getting out of it, it's not just what's theoretically possible, what's really happening, and all of these are important in that context. >> And I think, Arijit, that's amazing masterclass right there, soundbite right there, and I think it's because the data also is important, if you're going to be busting down data silos, you need to have a horizontally scalable data observability space. You have to have access to the data, so I think the trend will be more integrated, clearly, and more versatile from a platform perspective, it has to be. >> Absolutely, absolutely. >> Well, we're certainly going to bring you back on our conversations when we have our events and/or our groups around digital transformation Under the Hood series that we're going to do, but great voice, great commentary, Arijit, thank you for sharing that knowledge with us, appreciate it. >> My pleasure, thank you very much. >> Okay, I'm John Furrier with theCUBE, here, Leading with Observability content series with Splunk, I'm John Furrier with theCUBE, thanks for watching. (calm music)

>>law from Las Vegas. It's the Q covering a ws re invent 2019. Brought to you by Amazon Web service is and in along with its ecosystem partners. >>Welcome back to Las Vegas. Lisa Martin with John Ferrier, The Cube at AWS Reinvent 19 Lots of buzz. You can probably hear a little bit of it behind us here. There's about 65,000 people projected to be at a W s reinvent this week. Wow, we're very excited to welcome a distinguished guest and a distinguished architect from Splunk. Back to the Q r didn't murder, do you? Welcome back. >>Thank you very much. Thanks for having me back. >>Great to have you here. So let's kind of talk about here. We are re invent lots of news, lots of stuff. Lots of buzz going on. What kind of the latest with Splunk and a del us. >>All right, so the latest Splunk is obviously acquired us significance. The deal closed in trouble, So we're very excited about that. Um on we really feel that it's a it's a manager off complementary technologies, which is what I want some of the things we probably we can discuss later We're also very excited because we got acquired. Then we were able to go to dot com where we, you know, introduce the combined companies together. But then, at a cubicle on recently, we made a couple of very interesting product announcement that we're excited about, which is way discussing lots of reinvent conference. The 1st 1 is we have a brand new kubernetes experience called the community's Navigator, which we feel is a far, far better way Thio understand and make sense of the community environment. As you know, it's taking getting a lot of traction as a technology. So we're very excited about that because it not only gives you the infrastructure of you, but it also gives it the operators view, which I think operas. We really appreciate it. Three other thing that we're also focusing on. Obviously, if Splunk acquired US logs is an important part of this equation way are doubling down on the ability to ingest logs and make metrics out of them. You know, one of the things we've always discussed is how metrics every lightweight and actionable think that you can put on dashboard. You could put a lot son on the ability. Doing just logs and make them into metrics gives you that capability on the log data. We had a very interesting announcement around AWS. Fire lands on so on where you would be able to take love data from Splunk or other sources, and they bring them in as metrics to the system. The 13 has to do with the growing traction off open source standards. So we were actually very excited to make some contributions in the open telemetry project that we can discuss also later. But the idea is we want to promote open standards on open source, especially in instrumentation in the monitoring. Really? So that's kind of what's new >>question that's here at Amazon this week in this points to your success is observe ability, jazz he's laying out. This is distributed cloud Senator Gravity public Cloud Edge Outpost, Native AWS, Outpost five G with Verizon Wavelength All points to a lot of things. Move around, move compute to the edge where the data is so it speaks of large scale people having a hard time of doing it themselves on observe abilities. Harder and harder to roll your own are managed multiple tools. What are you guys doing to solve that problem? And how do you shape that going forward? >>That's a great question. Like the thing that blows my mind every time I come to reinvent is just the sheer variety of new things that comes across on. People are adopting them. All of these, he mentioned a bunch of different service is that I've got a lot of traction, got a lot of users, so that's happening across the user base. And then the question on D A. Y is because it's no longer about just building a database or, you know, things that you can sort some data and make some credit. It's about building the solution. A good solution. Need to support all the system. The service is that the customer the engineers are using right, so just keeping up with the sheer pace of innovation. Keeping that system up today is extremely, extremely hard. And so I feel that in generous making, less and less sense for most companies to try to roll their own observe ability, they would rather choose good tools that can sort of empower them that can able to move faster and invest in the people and process is part of it, which is also very, very key because >>the downside of rolling your own doing it yourself sure, what are some of the consequences that might happen? >>So in general, the people, the reason people want to build a couple of reasons, right? So one is they might undervalue, like the capabilities that good of the ruling might provide you, they might be afraid of the cost, like observe ability was cheap or free. Most people probably wouldn't build it. Some of them still vote because they might be afraid of vendor locking. Vendor lock in is a problem, and you don't want to be locked into vendors. Right? And what I feel in the terms of the risks is like if you consider observe ability as a cost center and not as an enabler, then you probably gonna try to do D i Y. But I think the view to the right view to have is think of it is something that accelerates your innovation and some of the risks of the advice. If you don't build something that's really capable that can that can do all the border or something that a system. Should you're gonna get slowed down, your innovation is gonna get slowed down. Another very thing, common pattern that we see a lot is maintaining, maintaining that it is a lot of resource is and people to build and maintain such a system. It's easy to prototype something and get it going, But are you going to be able to maintain the head count higher and grow the team on a long term basis? Because it's not something you can suddenly decide? Oops. I made a mistake. Time for a change. >>But change is difficult in any aspect of life. Changed management is something that we talk about office. It's way easier said than done. One of the things Andy Jassy talked about this morning and alluded to this and John's exclusive interview with him the other day was that the transformation needs to start at the top. It needs to be an executive level, a senior level and an aggressive tops down push in your experience in the last couple of years, what are some of the things that you're seeing companies in terms of the senior leadership embracing a understanding where D I y is useful where it's not, but also pushing that I want Oh my God, guys pushing it down from the top. So folks understand why this type of change is fundamental to a business to be competitive, >>right? So in general lighting, the focus is all on, like innovating, faster moving faster, keeping customers happy. Fundamentally, that's what we're doing. You know, our CMO Tom Bueller likes to say that you know the business. The Internet moves at the speed of life, a speed of life, Israel time, right? And so outages, Any kind of issues. They really affect your brand. And that's something that we need to avoid, like the plague, right? And that's gonna wear again. Observe. Ability comes in because this is the thing that's gonna allow you to find out renting There are. But more importantly, even when you don't have outages, the confidence that teams get in making changes, whether it be configuration changes or coat, which is a setup because they have a good system backing them up, is very, very critical. Right now. You can go D i y. You can go with a vendor solution, potentially terrifying, especially you can build one, but I think from top down. The important thing is like you have to be very clear about what you want out of it. And what are those things that you want to accelerate or make better in your organization? If your goal is, I want faster innovation, more code pushes, more changes, less deception like I feel that message needs to be done so that engineers understand that from management perspective, there's full support for this on their empowering you again. Where the two comes from is less important. But I think having those goals very clear and having that culture set from the top is very critical. >>A lot of open source discussions were hearing it here, laying out multiple databases you got pie towards you got tensorflow in machine, learning side on more and more kubernetes again, that's all speaks to where the service measures air going in. Micro Service's There's a lot of talk around open instrumentation open telemetry. What's your take on this? What is what's going on there? Can you share your commentary on those two things? >>Yes, so injured, as you know, like from the beginning, where since in Olympics started, we always believed that instrumentation should be open standards based. There should not be propriety instrumentation. They should be vendor lock in. It was a little bit perhaps ahead of the time, and we started off, but you can see that trend really accelerating now. But at this point, because of the sheer variety of service is and so on, it's very, very hard to build proprietary everything that supports all the all the things out there. What we're seeing is more bottoms up, open source, open standards efforts. Right, And that is great because A for the guys who are doing d i y. Because they don't want vendor lock in open standards is great because you're not really locked into a vendor in your environment. What you're doing is using a different back end, whether it be you know, your own or would it be a vendor's? Some of the things that we're doing is we're actually very happy to see this acceleration, and we're actually helping make that more so. Way just contributed pretty significant open telemetry project, which, as you know, is a way to instrument your environment for traces and metrics and logs eventually and so we actually donated the signal if Ickes smart Asian, which is pretty wonderful because it's a survey that's an agent that's running on your instances on your host, discovers as nuisances pop up. So, you know, speaking of community is the perfect fit for that, and it will start monitoring them and sending you did up on by making it by donating it to open telemetry. Were hoping to sort of accelerate out of the goodness and so that you know, all customers all use it. Whether they're significant customers or not should be able to benefit from that. >>Is an open source the source code? Or is it open as >>it's open source? There's two aspects to it is open standards as well as open. Both of them are happening because through the Amish in acquisition, we're now actually a pretty cool part of the open telemetry effort. So we're not really helping find finalize the standards, but also donating actual source code and components. >>Take a minute to explain. Signal FX is evolution now that you're in Splunk, right? What's changed? What's still the same? What's how is it? Evolve, how a signal effects evolved because you guys were really early ahead of it. A lot of people, but a lot of market power, great customer base and tech. What's the impact of Splunk and signal FX? >>Yes. So you know there's this cliche which is one plus one equals three. It kinda almost feels true here because, like I really, every time I think about this acquisition, it just feels how complimentary these two companies were because we have metrics and traces. Blanc has the best loss platform. But one of the things that we lot of times don't understand is he also a bunch of other technology which is highly relevant to the observe ability, space. For example, the acquired A company called Phantom, which is into automation, which is right up our alley because I feel like after all this mess has died down a little bit on communities, automation is gonna be the next frontier. They're fantastic. Automation platform built the security automation tool called Mission Control based on that, and now we're looking at how we can bring that into observe ability. Another example is incident management, Broncos Victor Ups, which is again exactly right up our alley. So we feel that we can really build a portfolio of solutions that work really, really well, that's one aspect. The other aspect, as you mentioned, is just the market power. And the resource is that's behind us, which is wonderful. For example. They're quite our mission, which is a fantastic complimentary technology to us, and we're working very quickly to sort of integrate the two together. Similarly, is getting the introductions. Having the financial benefit of a Splunk behind us is wonderful to have. So I think it'll only accelerate our >>congratulations on a great venture. I know you guys stayed the course and rightfully so great payday. But great outcome with Splunk Win is a win win. Yes, I gotta ask you the entrepreneurial question because a lot of people are saying, Oh my God, Amazon sucking up all the auction out of the room, Large scale. Got red shifts taking over this. That's taking over that someone's eating someone. Okay, I don't believe that. I believe that there's still a lot of opportunity for entrepreneurs because of this Born in the cloud and reborn in the cloud a new next gen architectures are developing with EJ. What's your opinion on this? As a cloud of alls What's the dynamics? And entrepreneurs and people thinking about innovating and either pivoting or reimagine their business? How should they be thinking about how to win in the new model? What are some of the architectural things that could bet on? What's your expert opinion on that? >>That's a good question. So I have some thoughts on it. Everybody might area once, right? So I feel like move to cloud is just happening. It's happened. Everything is going to move to the cloud. So I think the fundamental technologies like the databases, etcetera, that cloud provided they're always gonna have an advantage because they're going to be able to run it in a more performance way. But the thing that they're doing us a great favour are entrepreneurs is they're making a lot of different service is available to us now. They're not always necessarily all working well together to solve a specific use case. So I feel that they're giving us a tool set, among other things, to combined together to provide solutions for the problems that users organizations are facing. Not necessarily the platform but but the solution, the vertical on top of it. I think there's a lot of opportunity there, as well as sort of just new types of technology you can. As an entrepreneur, you can still build technology that the cloud provider might find as valuable, and they might want to buy you there right when I use you. So there's always opportunity there. But I think they're so busy building that the substrate, this enormous amount of opportunities for further up north. That's kind of my opinion. >>That's great opinion. >>Last question for you on the parlay of opportunity and the career that you've had as cloud is evolving the next gen of the cloud to Toto that John's calling it, and data becomes the critical element that can fine business differentiation and competitive advantages. What are some of the next industries you really think our prime to completely transform? If they get it right, >>I think we're still stop. It is a whole lot of talk of machine learning. I think we're just scratching the surface. I think what's happened is at this point it has become accessible enough on viable enough to be applied to different places. So every day we see a new headline where basically similar techniques were applied to this use case or that this case, and it's amazing being health care, transportation, you name it like digital business. It's happening all the way on our side, on our side of the fence. I feel a Splunk or a signal effects. We want to see a lot of that happening on our side of the fence, because again, because of the complexity, wonder thing that we have discussed with John earlier is how we feel machine learning and artificial intelligence gonna help us operate more efficiently because humans are going to be able to not really rock the entire complexity of what's out there. So I feel there's a lot of assistance that it can provide. That's one area which I think is interesting, And I feel also that one of the things we discussed within Signal FX is his move towards automation automated everything because complex systems, they just need to run themselves At some point. Humans cannot really go and make all the decisions like my my mainframe, itjust kind offer it to tell you we're not really in the middle of it, right to some extent. Similarly, I feel there's not a lot of action gonna happen on Automated Cloud and automated opposite really automated everything. So I think that's another sort of big area that I see happening on one thing that I also like to say that I don't want to make predictions because, like the world is so different from 10 years ago to now, it just blows my mind. I don't know whether I would have been able to sort of think what's gonna happen. So I only wonder what the next five years they're gonna >>bring. Love that opponent. You're >>right. Even a few years ago today, mine are just thank you for joining John A B on today. We appreciate your time. >>Thank you very much >>for John Ferrier. I'm Lisa Martin. You're watching the Cube from Reinvent 19 and Vegas will be right back.

(groovy music) >> From our studios in the heart of Silicon Valley, Palo Alto, California, this is a CUBE Conversation. >> Everyone, welcome to this special CUBE Conversation here in Palo Alto Studios for theCUBE. I'm John Furrier, host for theCUBE. We're here with special guest, Arijit Mukherji, who's the CTO of SignalFx, hot startup that's now growing very very fast in this cloud native world. Arijit, great to see you. Thanks for coming on. >> Great to see you too again, John. >> So cloud growth is changing the landscape of the enterprise. We're seeing it obviously, it's no real surprise, Cloud 1.0 has happened, public cloud. Cloud 2.0 as we're calling it is changing the game, where you're seeing enterprise cloud really the focus. We're seeing cloud native really move the needle. Kubernetes has kind of created that abstraction, kind of standard, defacto standard, if you will, people getting around. So you're seeing the game changing from how apps are built. >> Right. >> To security and everything in between. So a new set of services, web services at scale has certainly change the game. You guys are in the middle of this with monitoring and observability. And I want you to help us understand the core problem enterprises are having today because they know it's coming. They know it's here. They got investments out there. The cloud has changed the game for the enterprise, what's the problem? >> Yeah, so you're absolutely right, John. So everybody's moving to this sort of the new way of doing things, right. So monoliths are gone. Microservices are in, containers are in. And you're going to have to do that because we know if you don't do that, you're going to get lapped right, by the competition. And so the challenge right now is how do you make that successful? And the challenge there is these new environments are much much more complex as you mentioned. And the question is unless you understand how these systems behave, how will you be able to run them successfully? So the challenge as far as monitoring and observability is concerned is I think it's critical that it be there for it to be able to sort of do this cloud transformation successfully. But it's a far more complex and hard challenge than it used to be. >> We've seen the evolution. Yeah, we've been covering that for 10 years. It's the 10th year of theCUBE. We'll be celebrating that at VMworld this year. You've seen wave one, lift and shift, do some basic stuff, not a lot of heavy lifting. No tinkering with some of the tech in there. Monitoring is great. And then you've got rearchitect. Let's get some cloud native. Let's see some Kubernetes. And then the next path is this complete microservices. This is where everyone's really excited about. >> That's right. >> This is where the complexity is. So I got to ask you, that changes the notion of monitoring and observability. So given that this shift is happening, rearchitecting to full microservices, what is observability in that equation? >> Right, so there's a very interesting difference between I think, monitoring and observability that I think I would like to touch upon here. So you know, back in the days of the monolith, it was what we called classic monitoring. Monitoring is about looking at things, looking for things that you know might happen. So for example, if I know my server might fall down, I will run a probe to make sure that it's up or not, right? But when you move to this new world, I mean, have you, if you look at any cloud native environment with all the microservices and containers, for example Amazon's S3 has 120 different microservices powering it behind it. Now, if you were to go and ask an engineer like what is the map or how are the data flow happening in that environment? I guarantee you, no one person can probably do that well. So then, monitoring doesn't work because I don't even know what to look for. So what's important is I be able to gather telemetry, have the information available so that the unknowns, the kind of things that I'm not expecting because it's just too complex or just unanticipated. Like that data will allow us to figure out what went wrong. So observability is about gathering telemetry and information so that we can deal with that complexity, understand problems as they behave because the world is no longer simple anymore. >> So overall, observability is just monitoring in a dynamic environment, what you're saying. 'Cause monitoring used to be simple. You know it's going on. Static routes. >> That's right. >> Set policy, get some alarms. Network management, basic stuff. >> Exactly like Nagios checks and what not, yup. >> Now, you're saying there's unknowns happening, unexpected things going on around the services. What would that be just as an example? >> Yeah, so for example, again with microservices, why are we doing it? Because we want smaller teams to be able to innovate quicker, faster, right. So instead of my monolith, let's say I have whatever, SignalFx has 50 different microservices powering it. Now each of these teams, they are deploying software on their own because the whole idea of Cloud 2.0 is that we are able to move faster. So what that means is individual chunks of my overall service are adapting or changing over time or evolving. And so that's the complexity, like it's actually a changing landscape. Like my map does not stay the same on an ongoing basis. That is fundamentally a big challenge. The other challenge that I would mention too is that how ephemeral things are getting. So all these microservices that are themselves adapting, they're also being deployed in containers and by Kubernetes. Where these containers, they keep popping up and down all the time. Like even on infrastructure on which we are running it's extremely dynamic, right. Containers, Lambdas, sort of serverless is another great example of that. So it's a very shifting sands is what we're standing on, in some sense, right. >> And a lot of times, we cover a lot of real time. And you can't just throw in logs, you got to have that in there. This begs the question, okay, so I get the complexity. I'm a customer or I'm someone who wants to really go down this observability track with you. Why is it important? What's in it for me? >> So in the end, without it, how will you succeed? So it's almost like will a pilot with blinders on, will he be able to fly an aircraft? The answer is no. Similarly, I mean we may want to move to this modern awesome environment, which lets us move fast but unless you have visibility into it, unless you can find when problems are happening, unless you can, when those problems happen, be able to find the root cause and remediate them quickly, you're not really going to be successful. And so that's really why observability is important because it allows us to not only sort of run this well but it also allows us to understand the user experience because in the end, we are all service providers, we have users, right. And so understand what the user's experience is like. So that's important. Understand the key business metrics. If you look at a lot of the talk track that's been going around in the circuit around error budgets, and SLIs, and SLOs, which are sort of important things. The whole idea is that we want to measure and monitor what's important to the business, to the user. And that's kind of what observability allows us to get. >> You know gone are the days of a few application servers and a database. >> That's right. >> So on the why is it important, I got to follow up and say from an operations perspective, what is the new reality, okay? Because we know there's going to be a lot of databases out there, and a lot of different applications. You mentioned some of the containerization, dynamic microservices. But what's the impact and what's the importance to the operation side of the equation with observability? >> So what's happening now is again, back in the monolith days, the operators, the IT staff, who were running those infrastructure, they were the ones who would implement monitoring, right. But if you see the way now these environments are structured, these organizations are structured, it's the developers who are building tools. They are the ones who are also running them. And in order for an organization to be able to move fast, they need to give powerful tools to their developers to do their job. And because there is no one person who knows the right way of doing things. So it's really about sort of democratizing that capability. So you will need to give powerful observability tools to the developers, the operators, who are also the new operators, to sort of make with it what they will, in the sense that they are ones who best understand the meaning of the data that's being collected. Because it's all very specific to individual microservices. So that's really a powerful observability platform is one that allows you to easily collect a lot of information, allows you to analyze, visualize it, and sort of treat it in a way to sort of it helps you answer the questions you want to answer. >> So you're saying that okay, ops gets monitoring and observability. But a new persona, user is a developer. >> That is correct. >> And what do they care about? 'Cause they just want it to be abstracted away. They're not really probably wake up and say, "Hey, I can't wait to look at observability." So is it more of a use, so talk about the developer dynamic 'cause this is, that seems like a new trend. >> Yes, it totally is. So things are becoming less about black box testing, and more about sort of observability being an end-to-end process. So let me tell you what I mean. So back in the day, let's say, I implemented, I deployed a monolith. It was a Java server. There were standard ways to check them as a black box. I could run probes, et cetera, to run a health check end point and whatnot, life was great. But now, that's obviously not good enough. Because as I mentioned, because of interactions, because of complexity, a black box testing doesn't even work because like I said, the whole environment is very dynamic. So what the pattern now is that as I said, observability is an end-to-end process in the sense that developers care about observability when they're writing the software. It is not an afterthought anymore. So as I'm writing, as I'm developing software, I think about well, when this thing goes out into the wild, how will I monitor it? What are the things that I care about as a developer? Because I understand the system the best. And so you instrument, you build systems for observability is I think a big change that's happening. And once that happens, when you are the one person, who also is able to best read that data. >> So while they're developing, they get these benefits inherently right there on the spot. >> That is correct. >> This is kind of consistent with the live programming trend that's really popular in some languages. Rather than doing all the debugging, post event, coming back to it. >> That's right. >> So making it very efficient seems to be a use case. >> Yes, you are absolutely right. It's one of the things I'll actually talk about a lot actually is you know, observability, what is it for? Is it just for telling me when my production is not working well? The answer is no. Even when I'm developing, I may want to know well, did I have a performance degradation? How do I know that the code that I wrote is good? So I use again the same telemetry that I'm going to use later, even during the development process to make sure that the code that I wrote works well. We do the same thing during deployment. Again, I deploy a version or a canary or a few of them. Are they running well, right? So it is not just about what's happening in production, it's about end-to-end from development and deployment up to production. >> And that's what developers want. They want it in the moment, right when they're coding. >> That's right. >> Taken care of. >> It's instant gratification, like everybody else wants. >> And more efficiency. They know it's going to break, they know the consequences, they can deal with that. >> Yes. >> This is awesome, so the next question I have for you is how do you implement observability? >> That's a great question. So you can think of it as sort of in two ways. One is the means through which you get it. So you get observability through metrics, through logs, through traces, through probes, et cetera. That's one way. Another one is I think I alluded to a little bit earlier is what are your goals? Because everybody's goals are different, right? And if you think about in that sense, then the sort of the purpose of observability are a few. A, is it allowing your teams to move faster? So I spoke about some of the process just about earlier. Are they able to deploy code with confidence, faster? When problems happen, how quickly are we able to then triage them? So the whole incident review process. That's kind of important, observability better help me with that. The user experience is also something very important. As I mentioned, observability is going sort of more up the stack, so to speak. And so being able to understand what the user experience is, is very important. Similarly, understanding from the business point of view, what does the business care about? For example, when I had that outage, how much loss did I have? How many eyeballs did I miss on my side? So I think one way to think about it, you need to have good processes, good tools. At the same time, you need to be clear about what your goals are, and make sure that sort of whatever you're implementing, sort of furthers those to some extent. >> I'd like to play a little CUBE game here with you, and walk through the observability myths and reality. >> Sure. >> I'll say the myth, and you can tell me the reality. Myth number one, having observability reduces incidents. >> No, actually it might increase it. Let me put it that way, I'll tell you why. So it's almost like in a human, I may be measuring someone's temperature or pulse rate like every day. Does that make the person less or more prone to health problems? Chances are it's going to be the same. I might actually find things that I was not aware of, right. So in that sense, just having observability does not necessarily change anything about the process. But what it will do though is when a problem does happen, it I have this treasure trove of data, which I can then use to quickly isolate the problem. So what it does is it shrinks the outage time, which is in the end, what's very very important. So while it may not reduce your outages, it will definitely make them better from the end user point of view. >> Second myth, buying a tool means you have observability. Reality? >> No. Having a doctor doesn't mean I am healthy. In the sense that I think a tool is very important. It's a very very important step but the question is how well adopted is the tool? What kind of data am I sending to the tool? How are the users, my engineers, how well versed are they in using it, right? And so there's a lot of other stuff associated with it. So tool selection is very important but I think adoption and making it a success within the organization is also very very important. >> Okay, final myth, observability is free or cheap. >> I wish, so. >> Well you're a for-profit company. >> That's exactly right. No, I think, I really feel that observability is almost like a, it's an associated function. As I mentioned earlier, if I'm going to be successful flying this plane, I need commensurate amount of other services that sort of help me make that successful. So in a way, one way to think about it is it is it scales up as complex and as large as your environment gets and justifiably so. Because there's various other reasons too because in a way, adopting new technologies all the time. So tools are getting just more and more complicated. My requirements are getting more complicated. And then another thing I would add is the quality of my service like the level, the quality of service that I provide, the higher bar I want, I probably am spending more on observability. But it's a justified cost. So I think it's not a fixed cost obviously. It grows with your complexity and the kind of quality that you want to provide. >> Well, it's also, I mean I think the observability challenge with the complexity is there's a hidden cost though if you're not observing the right areas. >> Yes. >> The cost of not having that visibility, as a blind spot, could have business benefits, I mean not benefits, but consequences in a sense of outages, security, I mean there's a lot of different things that you got to have the observation space being enterprise. >> You totally have to do that. It's actually one thing we like to say is that instrument first, ask questions later. Now, coming from a for-profit vendor, it may sound sort of self-serving. But it's kind of not so too in the sense that I mean hindsight is 20-20. If I am stuck in a bad situation, I have no telemetry to sort of fall back on. Then where do I go with it? So I think we should be more conservative and sort of try and instrument the things that we think might be applicable, the kind of questions I may want to ask when a rainy day comes. So I think you're absolutely right. So it has to be something. It's a philosophy that developers, engineers, should sort of imbibe and they should then practice it as part of their sort of own workflows. >> I want to get into where company should invest in observability but I want to just throw a wildcard question at you, which is when you look at the big data space, even go back 10 years ago to now, cloud, there's always been they're talking about tool versus platform. And anything that's been data centric tend to be platform like conversations, not a tool. Tool can be like okay, it does a thing, does it really well, a hammer, everything's a nail with the hammer. But there's more dynamic range required 'cause you're talking about observation space, talking about cloud, horizontally scalable, hybrid on-premises. >> That's right. >> So again, it kind of feels like a platform technical challenge. >> You're absolutely right. So I think two factors at play if you ask my opinion. One is if you were interested in monitoring, a tool is perfect, right? Because you kind of know what you want. If a tool does that well, there's more power to it. But if you don't know what you want, if you are basically collecting stuff and you're depending on it as a way to answer questions on the unknown unknowns as Charity Majors like to say, then you do need a platform. Because that platform needs to be sort of inclusive. It must have data of different types, all be able to come into it. It's not really meant for a specific purpose. It's meant to be a generic tool. So we do see this trend in the industry towards more sort of a platform approach to this. Obviously, they will have tool-like capabilities because they're answering sort of particular use cases, et cetera. But the underlying platform, the more powerful it is, I think the better it is in the long run. >> Yeah and the argument there could be if it's an enabling platform, you can create abstraction layers for visualization. >> That's correct. >> Or APIs, other services. >> That's exactly right, yes. >> Okay, so now, I'm interested. I want to get started. How do I invest in good observability? I'm crossing the chasm, I'm going to full microservices. I've done a rearchitect, my team has got cultural buy-in. We're hiring, we're building our own stack, we're going to have on-premise, we'll be in the cloud. In some cases, fully cloud. What do I do, what do I invest? What's my play book? >> Sure, so I think we talked about the first one a little bit. So you have to choose the right tool. And the right tool in my opinion is not the one that does the best job now. When maybe I'm small, I'm not fully there yet. We have to think about what's the right tool or the right platform for when I'm, where for where I do want to go. Now, that may be commercial, that may be open source, that's not the point. But the point is that we need to have a very considered thought about what is it that we're betting the farm on, right. So that's number one. But that's not good enough. So as I mentioned, we need to make sure that there is a usage and understanding of the tool within the organization. So a big part of it is just around the practice and the culture of observability within the organization. So for example, good habits like every time we have an incident, you speak about these are the things that we measure, this is how we use our observability tool, here are the dashboards that we depended on. Sort of reinforcing those concepts over and over again so that those who are on board, they are obviously doing it right. Those who are not, they see the value and they start sort of using those good practices. That's kind of very important. The third thing I would say is start moving towards more higher level monitoring, observability. So measure the user experience, measure what's important to the business. Stuff like that are important. The fourth area that's sort of very very key is around sort of the whole incident management process. This is actually a very active topic. A lot of discussion going on out there right now is you know, it's great that I have this great tool, I have all this telemetry. I found there was a problem. But when that incident happened, there are a lot of again good practices. This is part of the whole culture and process of observability is how do we make that process smooth and standardized, and sort of become more efficient at it, right? And let's say you were to do that, the final sort of end goal is as we like to call is a self-running or a self-automated cloud, where do we really need humans in all of this? How can we sort of run remediation in a more and more automated fashion? At least for the stuff that maybe does not require a human intelligence, right? Actually, you'll find that 80, 90% of issues probably don't. If you think hard about it, don't require human. So I think this move towards automation is also another sort of very fantastic trend. I've seen that being very successful in the past. Some of my old companies. And I think that's going to be a trend in later stage. >> Yeah, for known processes, people and process. >> Yup. >> That's where problems come. Bad process or code and people mistakes. You can automate that. Mundane tasks or on differentiated kind of heavy lifting. >> That is exactly right because you know, there was this interesting study done that was commissioned by Stripe, where they found that among the CXOs, they value engineers and developers more than money. Because getting them, because they are such a scarce resource. So if you do get them, you probably don't want them to sort of run and do mundane things. That's not what you hired them for. So you want them to do the work of the business, and the more you can isolate them from sort of the mundane and they have automation come into play, I think it's just better across the board. >> Oh, they're investing more from the CSOs and the CIOs we talked to. >> Right. >> There's more investments in-house now for real development, real software development, real projects. And those top talent, they want to work on the toughest problem. >> That's exactly, that's why we're moving to a SASS future right? Because all the function that are not core to my business, I want to farm off and have somebody else take care of them. >> So you got me sold on observability, I'm a big believer in the observation space. And I certainly think cloud as they're horizontally scalable, elastic resource, and certainly, the Kubernetes trends with service measures and all the stuff going on, Kubeflow, and a bunch of other things. More and more services are going to be very dynamic. >> Yes. >> So you're going to have a lot of unknown and unusual patterns. >> That's right. >> That's just the way the internet works now. So you had me sold on that. Now, I need to get my team to the next level. They bought into devops. How do I take the temperature of where our IQ is in the life cycle of observability? Because I got to know where I am. Is there a way I can track my maturity or progress? >> Yes, yes, it's a topical question because I go out and I meet a lot of prospects and customers. And it's part of my job. And a lot of times, because sort of we are sort of in the leading edge, they will ask us, they look to us to tell them like are we doing it right? Or how did you guys do it? So just so that sharing of information, how can we get better? So as part of that, we actually, at SignalFx, we actually built a maturity model. It's a way for us to sort of evaluate ourselves across various dimensions to see how well are we doing? Not only, it's not just the score, it's also about how well are we doing? But how can we improve? Like if you wanted to go to the next level of maturity for example, like what are some of the things that we can do? And it spans multiple dimension. It starts with how are we even collecting data, right? How easy is it, in other words, for somebody starting a new service or using a new software to get to the business of observability? That's kind of important. You got the data, how will am I able to visualize it? Because effective visualization is as you can understand, very important. The next part comes with alerting. So well, things are running. I know how they're going. How well can I detect when problems are happening? How soon can I detect when problems are happening? What kind of items can I monitor? Can I monitor the low-level things? Or can I monitor the higher-level constructs? When the problem happen, let's talk about remediation. How quickly can I triage the problem to find out where it was? What kind of tools and slice-and-dice capabilities do I have? That's an important part of it. Let's say I did it. After that comes things like remediation. So I found the problem, how well can I remediate so like we talked about automation. So there is multiple different categories where we sort of, we talked about what, we've seen in the field in terms of what people are doing as well as some of the best practices. >> So you're going to make this tool available to customers? >> Yes, we have, it's actually available on our website. And if you come to our website, you'll be able to sort of run the assessment, as well as sort of see all of it yourself. >> Well, we've been following you guys since you launched. You've got a great management team, great technical chops, we've covered that. And this observability is a real trend as it moves into more complexity as we talked about. Most customers that are getting into this are trying to sift this from the signal, from the noise, and trying to think, decide who is the leader, and who is not. So how would you describe what a leader in enterprise observability looks like from a supplier's standpoint. You guys are one, you want to be the leader. You're the market leader. >> Right. >> What does a leader look like from a customer's standpoint? What are the things that have to be in place? What are the table stakes to be that leader? >> Sure, that's a great question. And yeah, so we did, SignalFx, we did build SignalFx to be a leader in this space, frankly. And there's a lot of different aspects that goes behind like what creates a good supplier in this space. One is I think you have to be open and flexible. Like you have to be, it's a platform play. You better be able to collect data from all the systems that are out there. The kind of the quality of the integration is very important. Another big thing we're finding is scale. A lot of these systems might not work when you move to sort of large numbers. And the problem that we are seeing is while I may have a hundred servers, I may be running 10,000 containers in those hundred servers. So now, everybody is a scale player, right. So the question is will your platform really be able to handle the complexity and the load? So that's an important one. Analytics as we mentioned is another very very important capability. I'll like to say that the ability to do analytics is not just good enough. How easy is it to use? Like are you developers and engineers, are they even using it? So the easy and the capability of analytics is important because that's kind of what allows us to measure those KPIs, those SLIs, those business metrics. And so that's kind of important. Slice-and-dice capability like how fast is the tool? Because when that outage happens, I don't want to run a five-minute query to sort of find some suspicion or you know to. And so the question is how quickly will it answer these ad hoc questions for me when the problem happens? So sort of the whole triage process that I talked about. The ability to support automation is one. The ability to, as I mentioned to take in different types of data, traces, metrics, be able to play with logs. All of those are sort of important aspects of it, yeah. >> Final question for you. If a customer says, "I'm going to cross the bridge "to the future with SignalFx." What's some of the head room? What's some of the futures that you would expect the customer to imagine or expect down the road as observability becomes more scalable? I can imagine the metrics are going to be all over the place. >> Yes. >> A lot of unusual patterns. New apps could come in that could be hits. And new data comes in. So as you take them today in observability, what's the next level, to cross that bridge to the future? >> Sure, sure. >> What's the next expectation? >> I think one thing will be to expect the unexpected. Because the world is changing so fast, I think you would probably be running things that you won't expect later. But a few things, I would say yes. So I think the proliferation of metrics and traces is like a big trend, where we see there used to be this dependence on sort of these monoliths and APM that sort of is transforming a little bit. There is this also this concept of using data science and artificial intelligence to come to bear on this space. So that's actually an interesting trend we see, where the idea is that it's hard because it's a complex system. It's hard for humans to define exactly what they want. But if the system can help them, can help identify things, that's actually really fantastic. Another one that we sort of briefly touched upon is automation or self-automated systems, where I think well, the time you're going to see that platforms like ours are going to help you automate much of this in a safe manner, because these are controlled systems, where if you, things can go awry and that's not a good position to be. So these are some exciting areas, where I think you will see some development down the road. >> And we've been seeing a lot of conversation around correlation and causation, and the interplay between those as these services are being stood up, torn down, stood up, torn down. You can look at the numbers all day long but you got to know causation, correlation. >> You bet, you bet because I think a lot of times, we naively think about this as a data problem, right? Where I find the kink in the graph, and if I go looking, I'll probably going to find a hundred different things that were sort of also correlated. Some of them may or may not be related to it. So I think a good tool is one that sort of gives you a sense. It sort of creates a boundary around the data set that it needs to look at, that is sort of relevant to your problem, and able to give you clues to causation. That you're exactly right because again, complexity is a hard problem to deal with. And anything that we can do to sort of help you short-circuit some of the pain is awesome. >> And I think you're on the right track with this developer focus because devops has proven that the developers want to code, build apps, and abstract away the complexity. And certainly, it's complex. >> That's right, that's right. It's fairly complex. >> Arijit, thanks for coming on. Arijit Mukherji, the CTO of SignalFx here inside the special CUBE Conversation breaking down the future of observability, where monitoring is going to the next level, certainly with cloud, impact to enterprise cloud. I'm John Furrier, here on theCUBE. Thanks for watching. >> Thank you. (groovy music)

>> Live from Las Vegas, it's theCUBE. Covering AWS re:Invent 2018, brought to you by Amazon Web Services, Intel, and their ecosystem partners. >> Welcome back here at the Sands, as we conclude our coverage here of day one of AWS re:Invent, we've been live on theCUBE, we'll be back with you again on Wednesday and Thursday, but glad you're here with us on Tuesday for our coverage, along with Justin Warren, I'm John Walls, and we're joined now from two executives from SignalFx, Karthik Rau, who's a CEO, and Arijit Mukherji who's the CTO >> Hi. >> At SignalFx, gentlemen thank you for being with us. >> Oh, it's a pleasure being on. >> Alright, so just tell us a little bit about what you do, and why you're here, and then we'll dive in from there. If you would. >> Sure, SignalFx is a cloud monitoring service, designed for operators of applications and infrastructure that might be running in the cloud. Our origins came out of Facebook, so Arijit and much of our technical team are responsible for building the monitoring systems in Facebook, back in the mid 2000's when they had their famous move fast and break things culture. >> Right. >> Which today everyone calls devops, and so what we've really focused on is building a far more analytics centric monitoring approach, that focuses a lot on identifying the patterns that are really meaningful and we believe that's a far more important problem to solve in today's distributive environments. >> And you made some news not too long ago, you've unleashed a new product into the marketplace, Arijit, if you would. >> Yes, yes, we are very excited to launch our, what we call a SignalFx microservices APM product. And it's really aimed at giving customers visibility into their transaction flow that's happening in their microservices environment. As you know, we're moving to microservices, the individual pieces are becoming smaller, and they're growing in number, and so the complexity of those interactions becoming harder and harder to manage, and this product is aimed, basically to help our customers make sense of those, and monitor them effectively. >> A theme that's come up a couple of times today on theCUBE, is that the complexity of the modern way of doing things, in cloud native services, microservices, it's beyond human comprehension >> That is exactly right. >> You need to have the assistance of tools like IPM, and I think we were talking just before we went live, that this is a distributive tracing type approach to microservices, is that correct? >> That is correct. So the goal is to have a lightweight approach, where you can very easily generate the spans, and traces from all your microservices, the whole environment, and the value that we provide is to sort of take them, baseline them to give you a sense of how performance is happening overall in the environment, but more importantly to your point earlier, is how can we help the customer using data signs to help them guide them towards the problem when it is happening, where it is happening, so that you know you can reduce the MTTR, which is sort of the key part of all of this, so that's been much of the focus of the product, yes. >> Okay, so for customers who are looking to re-platform onto microservices, or some of these newer ways of doing things, what is it about SignalFx that helps them to understand how to change an application from one way of doing things, you know monolithic type application, into something more microservices driven? How does SignalFx actually help them with that journey? >> Well our customers who are early in the cloud journey, are doing a number of things. One, they are able to get complete visibility into the old, right, so you typically want to look at a side by side, so you're able to leverage in our smart agent, collect information about your monolithic stack, get full visibility into what the performance looks like in that particular environment, but then what we do better than anyone else, is give you comprehensive visibility into the new stack, and give you the analytics that will allow you to really compare one versus the other, so one of the things that's very different about SignalFx, is we have a very rich analytics capability in the backend, so, collecting metric data across your environment, whether it's your old stack or your new stack, we're able to provide very sophisticated analytics to identify meaningful patterns, outliers, anomalies, and to look across all of your metadata to be able to identify whether those patterns are specific to a subset of machines or a particular version of code, and that's typically very helpful to customers as they're moving from the old to the new. >> Yeah, can you give me an example then, I mean, in terms of specificity that you've provided, you talk about sophisticated measurements, or stats, just something that would tell us, oh I see, that was kind of an aha moment, maybe for one of your customers. >> Yeah, so the thing that is unique about us is, that because we have a strong metrics product that's backing this, because we have a strong analytics capability that's backing this, when we do distributive tracing, we are tracing and providing you insight not only into your application, like what it is doing, we are actually able to correlate that with your infrastructure, so let's say your application is running in a container, if there is a problem we can actually let you correlate that application in the performance of that to the container, to the host, to the infrastructure, top down as well as sort of left right, so to speak, and that has been sort of key, because what we find is having that capability, really helps our short circuit, the resolution time, because a lot of times the problem may be vertical, other times it may be broad, like horizontal, right? So our goal is to catch both of them. >> Okay then. >> So you're able to identify the root cause of an issue much quicker, so your teams can go and find, that server's failing, I need to go and replace a piece of hardware, or there's a storage issue, and you can just dial it straight in really quickly, is that, is that-- >> Yes well. >> In modern environments, you're far more likely to see performance issues in a small subset of your transactions than you are to see just a massive outage, right? A lot of modern distributed systems are designed to be resilient to individual node failures, for example, in a future that we just launched along with our microservices APM, is something called outlier analyzer, so let's say all your metrics indicate the service is performing fine, but you have point five percent of your users complaining that the performance is terrible. That's where tracing really helps, 'cause now you can look at every transaction, you can understand exactly where, you know, things might be slow, but it's typically a trial and error process, you have to go through every single trace you have to sort of figure out is it a particular version of code, or particular server. Our outlier analyzer feature will automatically look through all of the outliers, identify the over represented dimensions, and guide you to those specific problematic areas, right? So you run our outlier analyzer, it'll tell you, you know, this particular machine is overrepresented in your long tail traces. >> Yeah. >> Or this particular version of code is overrepresented. So, it short circuits the entire troubleshooting process by orders of magnitude. >> Yeah, that kind of intermittent error is always really really hard to find, something which just explodes and catches on fire, that's easy to find. >> And it's extremely difficult for a human to find it by trial and error across a distributed system, that can involve thousands of components right, so you really really have to leverage analytics and that's really what SignalFx is incredibly strong at doing. >> Yeah, so we're basically replacing luck with tools. >> Trial and error and luck, with a more prescriptive trouble shooting. >> Yeah, so for customers who have gone through this journey, and they've actually re-platformed an application, they've converted it into microservices, and they're doing cloud native things, and you've helped some of these customers. What's an example of a customer who's living in that new world, or what's the view like from where they're sitting, where they have all these lovely tools, and they're not relying on luck anymore, what's their sort of daily life like? >> Well I think the biggest difference is, they're now able to automate a lot of remediation, if you can be more intelligent in the signals that you're capturing, apply more intelligent analytics, then you, especially in today's environments, you can automate a lot of remediation, today's frameworks are highly automatable. And so one example of this, we have one of our larger Fortune 500 accounts, they do a number of launches, product launches, where they get massive amounts of load during a product launch and this is not atypical in today's environments. >> Yeah. >> And prior to having the real time data collection analysis with SignalFx, they would have two rooms full of people, supporting every single launch, and very reactively, you know, and something would go wrong they would have to go and figure out what was happening. With SignalFx they are now able to build very sophisticated analyses on the data as they're spinning up containers and instances to support a shoe launch, and they've now actually automated a lot of their remediation, whether it's auto-scaling, or rolling back of you know canary releases and such, and they've gone from having two rooms full of people to having just one on call engineer every time they do a launch, and it's also enabled them to be a lot more aggressive in doing these launches because they just have a lot more confidence in their ability to execute them. You know, that's one example. >> You know Justin was talking about some of the trends, we've heard a lot about today, the one I guess, or one of the constants has been about the pace, the rapidity of innovation, the rapidity of change, and so in your world, what do you think is the next hate to say big thing, but what mountain are you trying to climb now, that you haven't already conquered? >> So in my view, there's some very very encouraging trends that are coming our way actually, there was a talk that I presented earlier today about the concept of service measures and how I feel that they are going to be the next big thing because I think they attack a lot of the core operational challenges that we face in our microservices environment including how well you can instrument your environment, how well do the different types of instrumentation, your metrics, your APM, your logs, how well are they relatable, how tightly coupled are they? Right, how quickly can you make configuration changes within the environment in a more foolproof manner, that's more automated, that is more consistent, and so I feel like technology like that is going to transform how we do software in a few years from now, I see that advancing very very quickly, and something that's very related to that, and something I eluded to, to the talk earlier too was, this concept of feedback driven automation, where now I am no longer just going and just configuring my infrastructure to behave the way I want it to. In fact I'm also observing it as it is running, using high quality monitoring tools like SignalFx, and then using that to create new feedback, because if things sort of diverge from my intent, that I should be able to get it back to where I want to be, and all of this must happen without human interaction, because we work in the order of minutes, while you know automation can do this in seconds. This is absolutely fascinating, I think this is one of those big trends, that are coming down the pipe. >> Karthik anything to add to that? >> No I think Arijit nailed it. (laughing) >> Excellent, alright. Gentlemen thanks for being with us. >> Thank you. >> Good luck with the rest of the show, I'm sure it's been very good for you so far, and for the next two days, have a great time. >> Okay. >> Thank you very much. >> Excellent, thanks for being with us. >> We are concluding our coverage, day one, here of AWS reinvent for Justin Warren, I'm John Walls, we thank you for watching theCUBE. (upbeat music)

>> From Union Square in downtown San Francisco, it's theCUBE covering PagerDuty Summit '18. Now here's Jeff Frick. >> Hey welcome back everybody. Jeff Frick here with theCUBe. We're at PagerDuty Summit at the Westin St. Francis in Union Square, historic venue. Our second time to this show, there's about 900 people here talking about kind of the future of dev ops, but going a lot further than dev ops. And we're excited to have a couple of CUBE alumni here at the conference from SignalFX. We've got Arjit Mukarji. >> Mukarji, yeah. >> Thank you. And Karthik Rao, co-founder and CEO of Signal FX. Gentlemen, welcome. >> Thank you very much. >> So what do you do at PagerDuty Summit? >> Well we've been partners with PagerDuty for a long time now, we've known them since the very early days, we share a common investor. But we both operate very squarely in the same space, which is companies moving towards dev ops development and deployment methodologies, leveraging cloud and native architectures. We solve a different part of the problem around monitoring and observation and we partner with them very closely around incident management Once a problem is detected, we typically integrate in with PagerDuty and trigger whatever incident management paths that our customers are orchestrating by PagerDuty. So, it's been really an integral part of our entire work flow since we started the company. So we're very close partners with them. >> Yeah, it's interesting 'cause Jen announced they have 300 integrations or 300+ integrations, whatever the number is, and to the outside looking in, it might look like a lot of those are competitive, like there's a lot of work flow and notification types of partners in that ecosystem, but in fact, lots of different people with lots of different slices of the pie. >> That is good. >> Yeah, absolutely. It's a really big problem space that everyone is trying to solve in this day and age. Some of our competitors are in that list, but you know we partner very closely with PagerDuty. As I mentioned earlier, our focus really is around problem detection and leveraging the most intelligent algorithms, statistical models in real time to detect patterns that are occurring in a production environment and triggering an alert, and typically we're integrating in with PagerDuty and PagerDuty deals with the human elements of once something has been detected, how do you manage that incident? How do you router to the appropriate people? One of the things that's really interesting as this world is changing towards these dev ops models is the number of people that have to get involved is substantially greater than it was before. In the old days, you would have an alert go into a knock and you have a specialist group of people with very specific runbooks because your software wasn't changing very often. In today's world, your software is changing sometimes on a daily basis, and it could be changing across dozens of teams, hundreds of teams in larger organizations. And so, there's a problem on the detection side because companies like SignalFX have to do a really great job of detecting problems as they emerge across these disparate teams, across a much, much, much, larger environment with much larger volumes of data and then companies like PagerDuty really have to deal with a far more complex set of requirements around making sure the right people get notified at the right time. And so they're two very different problems and we're very happy to- and have been partnering with them for a number of years now. >> And again, the complexity around the APIs where the app is running, there's so many levels now of new complexity compared to when it was just one app, running on one system, probably in your own data center, probably that you wrote, compared to this kind of API centric multi-cloud world that we live in today. >> That is exactly right because what's happening is our application architectures are changing 'cause we used to have these monoliths, we used to have three tiers and whatnot, and we're replacing that with the micro-services, loosely cabled systems, and whatnot. At the same time, the substrate on which we are running those services, those are also changing. Right, so instead of servers, now we have virtual machines, we have cloud distances and containers and pods and what-have-you. So in a way, we are sort of growing below too in some sense and so that's why sort of monitoring this kind of complex, more numerous environment is becoming a harder challenge. We're doing this for a good cause, because we want to move faster, we want to innovate faster, but at the same time, it's also making the established problems harder, which is sort of what requires newer tools, which sort of brings companies like us into the picture. >> Right, yep. And then just the shear scale, volume, number of data that's flowing through the pipes now on all these different applications is growing exponentially, right? We see time and time again, so it really begs for a smarter approach. >> Absolutely, I mean on two levels right? The number of minutes of software consumption is up exponentially, right? Since the smartphone came out in 2007, you've got billions of people connected to software now, connected all the time, so the load is up order sum magnitude which is driving, even if you didn't change the architectures, you would have to build out substantially more back-end systems, but now the architectures are changing as well, where every physical server is now parceled up into VMs which are parceled up into containers. And so the number of systems are also up by order sum magnitude. And so there's no possible way for a human to respond to individual alerts happening on individual systems, you're just going to drown in noise. So the requirements of this new world really are, you have to have an analytic spaced approach to monitoring and more automation, more intelligence around detecting the patterns that really matter. >> Right. Which is such a great opportunity for artificial intelligence, right, a machine learning. And we talk about it all the time, everyone wants to talk about those, kind of as a vendor-led something that you buy. Yeah, that's kind of okay, but really where the huge benefit is, companies like you guys and PagerDuty using that technology, integrated in with what you deliver on your core to do a much better job in this crazy increasing scale of volume that's run with these machines. >> Yes, because the systems are becoming so complex that even if you asked a human to go and set up the perfect monitoring or perfect alerting, et cetera, it might be quite a hard challenge, right? So, as a result sort of automation, computer intelligence, et cetera needs to be brought in to bear, because again, it's a more complex system, we need higher order systems that have dealed with them. >> Right. >> You are very, very right, yes. And that's a trend we are starting to see within the product, we are actually focusing a lot on sort of data science capabilities which too are sort of making them more and more sort of machine running and automation. In the future, we have capabilities in the product that can look at populations and identify outliers, look at cyclical problems and identify outliers again. So the idea is to make it easy for users to monitor a complex system without having to get into the guts, so to speak. >> Right. >> And to do it on various sorts of data, right? I think you have an interesting use case that we've been experimenting with recently. >> That's right. >> If you want to talk about that. >> Yeah, so I actually have a talk tomorrow, it's called "Interesting One." It's about monitoring social signals, monitoring humans. So we have these systems, we have these metrics platforms and they are quite generic, the tools that we have nowadays and are sort of available to us are quite powerful, and the set of inputs need not be isolated to what the computers are telling me. Why not look at other things, why not look at business signals? In my case, I'm going to talk about monitoring what the humans are doing on Slack as a way for me to know whether there's something of interest that's going on in my infrastructure, in my service that I need to be aware of, right? And you'll be shocked how surprisingly accurate it tends to be. It's just an interesting thing, and it makes one wonder what else is out there for us to sort of look at? Why confine ourselves, right? >> Right. It's funny because we hear about sentiment analysis in social media all the time, but more in the context of Pepsi or a big consumer brand that's trying to figure out how people feel. But to do it inside your own company on your own internal tool, like a Slack, that's a whole different level of insight. >> You'd be surprised at the number of companies that monitor Twitter to understand whether they have an adage. >> That's right. >> Yeah, because in this day and age, users are on Twitter within seconds if something is perceived to be slow, or something is perceived to be down, they're on Twitter. So there are all sorts of other interesting signals to potentially pull from. >> Right, right. Well and guess what, we were just at AT&T Spark yesterday and the 5G's coming and it's 100x more data'll be flowing through the mobiles, so the problem's not going to get any smaller any time soon. >> No. >> Absolutely. >> So what else have you guys been up to since we last spoke? Continuing to grow, making some interesting moves. >> Absolutely- >> Crossing oceans. >> We've been very, very busy, one of the big areas of investment for us has been international growth, so we've been investing quite a bit in Europe. We have just introduced an instance of our service that's based in a European data center. For a lot of our European-based clients, they prefer to have data locality, data residency within the European Union, so that's something new that we just introduced last month, continue to have a ton of momentum, outed AMIA, they're very much on the cloud journey, and embracing cloud and embracing dev ops, so it's really great to see that momentum out there. >> Right, and clearly with GDPR and those types of things, you have to have a presence for certain types of customers, certain types of data. Anything surprising in that move that you didn't expect or? >> No, I don't know, I'll let you. >> Not in that move, but it's just interesting to see how quickly some of these modern technologies are getting adopted and how- one of the things sort of we talk about a lot in our trade is ephemeral, right? So how things are short-lived nowadays, and you used to lease these servers that used to stay in your data center for three years, then you went to Amazon and you leased your instances, which probably lived for a few months or a few days, then they became containers, and the containers sometimes only for a few hours or for- you know. And then, if you think about serverless and whatnot, it's in a whole different level, and the amount of ephemeral that's going on, especially in the more cloud native companies, was a little bit of a surprise in the sense that, it actually poses a very interesting challenge in how do you monitor something that's changing so fast? And we had to have a lot of engineering put in to sort of make that problem more tractable for us. And it continues to be an area of investment. That to me, was something that was a little bit of a surprise when we started off. Much of this doctorization and coordinating was not yet in place, and so that was an interesting technical challenge as well as a surprise. >> Well I'm curious too as instances, right so there's the core instances that are running core businesses that don't change that much, but it's a promotion, it's a this or that, right? It's a spin up app and a spin down app. Are those even going up on the same infrastructure from the first time they do it to the second time they do it. I mean, how much are you learning that you can leverage as people are doing things differently over and over again as their objectives change, their applications change, they're going to go to market around that specific application. That's changing all the time as well. >> Yeah, so I think the challenge there is to sort of build, at least from a technical point of view, from SignalFX point of view, is build something that is versatile enough to handle these different use cases. We've got new use cases, new ways of doing things are going to continue to happen, probably going to keep on accelerating. So the challenge for us is good and bad, is how do we make a platform that is generic, that can be used for anything that may come down the pike, not only just now. At the second time, how do we innovate to continue to be up to speed with the latest of that's what's going on in terms of infrastructure trends, software delivery trends, and whatnot. Because if we're not able to do that, then that puts us sort of behind. >> Right, right. >> So it's a sort of lot of phonetic innovation, but it's also exciting at the same time. >> Right, right, right. And just the whole concept too, where I think what's best practice quickly becomes expected baseline really, really fast. I mean, what's cutting edge, innovative now unfortunately or fortunately, that become the benchmark by which everything else is measured overnight. That's the thing that just amazes me, what was magical yesterday is just expected, boring behavior today. Alright good, so as we get to the end of the year a lot of exciting stuff, you guys said you're going to be at Reinvent, we will see you there. Anything else that you're looking forward to over the next couple months? >> Just, we're really excited about Reinvent's big show for us, and we'll have some good announcements around the show. And yeah, looking forward to just continuing to do what we've been doing and deliver more rally to our customers. >> Love it, just keep working hard. >> Yep. >> Alright. Arjit, hope your throat gets better before your big talk tomorrow. >> Yeah, that's right. >> Alright, thanks for stopping by Karthik, it was great to see you. >> Great to see you. >> I'm Jeff, you're watching theCUBE, we're at PagerDuty Summit at the Westin St. Francis in San Francisco. Thanks for watching, see you next time.

>> Announcer: Live from the Moscone Center. It's theCUBE! Covering AWS Summit San Francisco 2018. Brought to you by Amazon Web Services. (upbeat techno music) >> Hey, welcome back, everyone. We're live here in San Francisco. This is theCUBE's exclusive coverage of AWS Amazon Web Services Summit 2018 with my co-host Stu Miniman. We have two great guests. Hot startup from SingleFx, the CEO, Karthik Rau, and the CTO, Arijit Mukherji. Welcome to theCUBE. Good to see you again. >> Karthik: Yeah, great to see you again. Thanks for having us. >> So, we've been following you guys. You've been out five years. Two years in stealth, three years ago you launched on theCUBE. >> Karthik: Right here on theCUBE. >> We see you at AWS and VMware. Cloud's changed a lot. So, let's get an update. Karthik, take a minute to explain where you guys are at now company-wise, employees, traction momentum, product. Where are you guys at now? >> Karthik: Yeah, absolutely. So, SignalFx, first of all, let me tell you what we do. SignalFx is a realtime streaming operational intelligence solution. Basically, what that means is we collect monitoring data, operational data across the entire cloud environment, from the infrastructure all the way up to the applications. And we apply realtime analytics on that data to help people be a lot more proactive in their monitoring of these distributed environments. We launched the company in 2015. We come ... I'll let Arijit talk about our origins. We came out of Facebook. And we had a lot of experience building this to Facebook. In the past three years, we've been building up our company aggressively. We've now got hundreds of customers including several large Fortune 500 accounts, large web scale accounts like Acquia and HubSpot and Yelp and KAYAK. And we're over 100 employees now, about 120 employees. And yeah, doing great. >> So, Werner Vogels, the CTO, laid out on stage plus a great Matt Wood conversation about machine learning but the real thing that Werner laid out was the old way, the web server, multi-tier architecture stack kind of thing going on there to a more cloud DevOps horizontally scalable where sets of servers that could be spawned in parallel creates a new kind of operating model but also creates challenges around what to instrument. You know, as we would joke, someone left the lights on, implying EC2s been running. And all these kinds of things are going on. And you mentioned some of the Facebook kind of challenges. People were building their own scale. What have you guys learned and how does that apply today's modern infrastructure? What are some of the threshold challenges that companies are facing when they say, one, already there or I want to get there? How do you guys look at the main issues? >> Karthik: Do you want to take that? >> Yeah, so monitoring modern environments and infrastructure is actually quite a challenge. There's obviously a few things going around. One, as you mentioned, is the variety, the sheer variety of things. No longer just the three-tier architecture I have cloud services. I have containers. I have lambdas. I have my own applications. I have the cloud infrastructure itself that all needs to be monitored. And things are also becoming far more numerous. So, there's just many more of everything, right? And so, making sense of that space is becoming a big challenge. And our company was founded on the idea that monitoring is becoming an analytics problem. So, it's no longer about looking at individual servers or applications instances. It's more about making sense holistically over what's going on and being able to combine different types of data from different systems together to provide you with that high level view and that's the kind of functionality that we at SignalFx have been trying to provide. >> What are some of the data flows volumes look like. Cause I've heard multiple people talk about either Facebook or in open compute environments where there's just so much data coming in from the instrumentation that no human could actually get their arms around it. And you need to supplement it with machine learning and intelligence. I mean, is that something that you're seeing? What are some of the -- >> Yes, so actually what we see is different prospects or customers will be in different stages of a spectrum where maybe they were in a stage one where they're sort of using traditional architectures and then moving to these more modern systems. And as they get more modernized themselves, their use cases or the ways they wanted to do monitoring also gets more advanced. And so, we see the whole spectrum of it, as you mentioned. And so, understanding analytically how what we're is doing is great. But then you also want to take the human out of things as much as possible, right? >> Yeah. >> And make things more automated. And you want to look at the data and how things are behaving to learn from existing patterns to find outlines. So, that's really a very interesting challenge. And what I look at what we can do as a company going forward, like all the technological stuff that we can invest in, it's quite interesting. >> Yeah, Karthik, take us inside your customers. How does this modern monitoring, how does it change their business? How does it impact things like feedback loops and DevOps and everything that customers are having to deal in this kind of ever changing environment? >> Yeah, well I'll give you an example. There's a Fortune 500 company. They do product launches. And this is one of our customers and their product launches drive so much traffic that they do 80% of their business in the first two minutes of a product launch. And this is not at all uncommon in today's economy. And they're leveraging a lot of modern technologies, container architectures, serverless function architectures to spin up a bunch of capacity during these launches. And they were effectively flying blind most of the time. Because most of the traditional systems management monitoring solutions are not designed, A, to handle that volume. But, B, to handle the instant discovery requirements of if you're going to do 80% of your business in the first two minutes. So, the challenge is you're always playing defense. You're reacting to issues. And you're mostly flying blind. By leveraging SignalFx, they're getting realtime visibility, realtime discovery of these components as they're coming up. We're the only solution that can do that. So, literally within seconds of spinning up all of these containers, they're getting live streams into their dashboards, and live analytics, and live alerts. And what that's enabled them to do is be a lot more aggressive and effectively doing a lot more of these launches. So, that's driving their business and it's helping them drive their digital strategy forward. >> And microservices is really enabling you guys to be more relevant. Because truly the signal from the noise is where all these services reporting to? >> Karthik: Yeah. >> You talk about container madness. >> Karthik: There are two fundamental problems. So, one there's an architecture shift. And that's driving massive amounts of volume. You have physical machines that will live for three years in a data center. Divide it up into VMs, 10, 20 VMs per server. That'll maybe live for a few months. To now every process running in it's own container that might live for a few minutes. So, you have a massive exponential explosion in the number of components. But that's not the only problem. I was part of an architectural shift at VMware for a number of years. We weren't just affecting an architecture change. What's happening now is there's a cultural change and a process change that's happening as well. Because with containers, your development team can push changes directly out into a production environment. And what you're finding is you're going from sequential product development to parallel product development and a massive exponential increase in the number of code pushes. The only way you can operationalize that is you have to have realtime visibility in everything that's happening. Otherwise, the left arm doesn't know what the right arm is doing. >> John: And you need prescriptive and predictive analytics. >> Exactly. And you need predictive analytics to identify there's something unusual here. It's not a problem yet. But this is highly unusual and maybe it's your canary release. We need to do a code push. So, you want to roll it back. So, having that level of predictiveness becomes absolutely critical. >> Yeah, you mentioned realtime. We used to argue what really is realtime. And it was usually well in time to react to what the customer needs. What does realtime mean to your customers? Architecturally, is there something you do different to kind of understand what that means? >> Arijit: Yeah, so we actually fundamentally took a very different approach when we build a product. Where, typically, monitoring our metrics, monitoring was done with what we call a store and create or a batch-like architecture where you store all the data points that are coming in, then you create from it to any other use cases. While what we build at SignalFx is a fully end-to-end streaming architecture which is realtime. And what we mean by realtime is like two to three seconds between a data point coming through us and it's firing an alert or showing up in your chart. So, that's the kind of realtime. And it requires us to do lots of innovations up and down the stack. And we've built a lot of IP. We've got now patterns. And more are coming because the approach we took was quite novel. Different from-- >> John: You guys got a great management team. And looking at what you guys have done. I've been impressed with you guys. I want to just ask, Karthik, you mentioned about all these parallel processes that are going on. Totally agree. The process change, operationalizing an all new cultural way to create software manage the data. I mean, it really is the perfect storm for innovation. But also, it could literally screw people up. So, I got to ask you, who are you targeting for your customer? Who is the person that you talk to? Assuming it's kind of DevOps, so it's more like a cloud architect. Who do you target? Who do you sell to? Who's the buyer? Who uses your service? >> Karthik: Well, we see ... Every enterprise we see following a very similar journey. So, the first stage is, typically, you're just getting familiar with cloud. And you're probably just lifting and shifting enterprise workloads into the cloud. Probably experimenting with big data on the cloud. You're not yet doing microservices or containers or DevOps. And for them, we're still selling largely to classic IT. There just trying to get better visibility into their digital environment, you know, they're cloud environment. But then, what ends up happening is they very quickly get to what we call basically chaos. It's stage two. And it has a lot of parallels to shadow IT. What happened with SAS, where you have hundreds of different SAS tools is happening all over again with cloud but you've got hundreds or thousands of different operational tools. Different ways of doing monitoring, logging, security. And every team is doing it's own thing. And so, that's a big problem for enterprises who are trying to build best practices across their broader team. In that place, we're typically selling to departments because they don't have a centralize strategy yet. But what we find is the organizations at maturity have figured out that it's important to have certain centralized core services. And that doesn't mean they're forced on the end users. But they provide best practices around monitoring, logging, and such. And just make it easy for them to use those solutions. So, that's almost a new IT organization. It's platform engineering -- >> John: Is that a cloud architect? >> Platform engineering team, infrastructure engineering team, and they are effectively building best practices around the new stack not the traditional stack. >> So, you are or aren't targeting department level? Are you are? >> We sell to departments. But we also sell to the teams that are standardizing across the entire organization. >> So, cloud architects, for instance? >> Depends on the stage of the cloud journey. >> Or company. >> And the company, exactly. >> From an architectural standpoint, you talked that there's virtualization, there's containers, now serverless. How do you even figure out what to monitor in serverless? How fast is that changing? And how is that impacting your road map? >> So, serverless brings a very interesting challenge because they are very, very ephemeral. Like they're ephemeral in some sense. So, we realize there are two things. One is serverless, there's a reason why things are moving faster. It's because you want to be able to move faster. But then you also need to be able to monitor faster. It's no good monitoring serverless at five minutes later, for example. So, one of the things we invested in was how to get metrics, etc. and telemetry from these serverless environments in a very fast fashion. And that's something that we've done. The second thing we are doing that really works for this environment is afterall it's not about how many times a serverless function ran, it's about the value that it's providing the application that's running on it. And by focusing on a platform that let's you send these application metrics in great detail and then be able to monitor and analyze them, I think really amplifies the value in some sense. So, those are the two ... >> John: And talk about the ecosystems. One of the things I want to ask you guys because we've been seeing a collision between a lot of the different clouds. Clients want multicloud. Well, obviously, we're here at Amazon. They believe they should be the only cloud. But I think most customers would look at either legacy systems with some instrumentation and operational data to edge of the network, for instance. I mean, look at the edge of the network. That's just an extension of the data center depending on how you look at it. So, how do you guys view that kind of direction where customer says, "Hey, you know, I got a cloud architect. We're on Amazon. Of course, we have some old Microsoft stuff. So, we've got Azure going up there. We're kicking the tires on Google. And I got this whole IoT Edge project. SignalFx, instrument that for me. (laughs) Is that what you do? Or how do you deal with that? How would you deal with that kind of conversation? >> Well, I think most enterprises, the larger companies we see looking at multiple clouds. And they have different workloads running in different clouds, depending on the needs and what they're looking to do. So, the nice thing about a solution like SignalFx is we span all of these different architectures. And what we find is that most of the larger companies want to separate their business process solutions from their runtime architectures. Because they want to have a solution like SignalFx that it doesn't matter who you're using. If you choose to have your analytics intensive workloads in Google Cloud and your eCommerce workloads in Amazon, but you only want one system that will page someone in the middle of the night if there's a problem, then you have SignalFx to do that. And then you have your choice of runtime environments depending on what your developers need or what the business demands. We provide a lot of that glue across the different environments. >> Do you see that as the preferred architecture with most customers? Cause that makes a lot of sense. I mean, whether you're doing other data services, it kind of makes sense to separate out. Is that consistent? >> To have different applications >> Yeah. >> In different clouds? It depends. I mean, I think we see some people who are more comfortable running on a single cloud vendor and they make the decision based on what a portfolio of platforms and service features that are available. And they really like those, and they say it's easy to just go with one. But more often, we find people wanting to at least have some percentage running in a different cloud vendor. >> John: All right, final question. What's the secret sauce for the company? Tell us about the secret sauce. >> Arijit: I think-- >> We got the patents. I heard patents. You don't have to show all this exactly. But what is the secret DNA of the tech? What's the magic? >> I think it's our very unique architecture. It's entirely different from what you have. It's streaming and it focuses on scale, on timeliness, as well as on analytics capability. I think that unique combination is very special for us. And that, in a way, sort of allows us to address very, very different use cases, including this hybrid environments and what not, in a very effective way. So, it's a very, very powerful platform that can be used for many use cases. >> All right, so that was John's final question. Karthik, I've got one last one for you. What's it like being a CEO of a software company in the cloud era today compared to what it's been earlier in our career? >> Well, it's moving very, very quickly, right? I mean, technology always move very quickly. But I think compared to when I was at VMware in the mid 2000s, it just feels like every 18 months there's a new technology wave. You know, when we started our company five years ago, that was the first year that AWS eclipsed a billion dollars in sales and Dagra hadn't even launched. It launched a month after we started the company. And then serverless came. And now function architecture is all there. So, there's just so much change happening, and it's happening so quickly, it forces vendors like us to really be on the cutting edge and forward looking and making sure that you're keeping an eye out for what's coming cause the markets are moving way faster, I think, then they were 15 years ago. >> John: Well, Karthik, thanks so much. We appreciate you guys coming on, SignalFx. I'll give you the final word on the interview. Take a minute to share something with the audience that they might not know about SignalFx that they should know about. >> Well, I think what people may not realize is how realtime we can actually get. I think most people are used to doing all their monitoring and observation, and they think of realtime in the order of minutes, or if you can get stuff every 30 seconds. We really are the only realtime solution. That's why we say real realtime. We're on the order of seconds. You can build really, really sophisticated analytics and get visibility like you can't anywhere else. So, it's real, realtime. >> And that's soon to be table stakes. TheCUBE is realtime. We're live right here, on theCUBE here, in San Francisco at Amazon Web Services, AWS Summit 2018. We've been covering all the Amazon re:Invents since it started, of course. I'm John Furrier with Stu Miniman. Back with more live coverage after this short break. (upbeat techno music) (gentle instrumental music)