>> TheCUBE presents UI Path Forward Five brought to you by UI Path. >> Welcome back to UI Path Forward Five. You're watching The Cubes, Walter Wall coverage. This is day one, Dave Vellante, with my co-host Dave Nicholson. We're taking RPA to intelligence automation. We're going from point tools to platforms. Neeraj Mathur is here. He's the director of Intelligent Automation at VMware. Yes, VMware. We're not going to talk about vSphere or Aria, or maybe we are, (Neeraj chuckles) but he's joined by Thomas Stocker who's a principal product manager at UI Path. And we're going to talk about testing automation, automating the testing process. It's a new sort of big vector in the whole RPA automation space. Gentleman, welcome to theCUBE. Good to see you. >> Neeraj: Thank you very much. >> Thomas: Thank you. >> So Neeraj, as we were saying, Dave and I, you know, really like VMware was half our lives for a long time but we're going to flip it a little bit. >> Neeraj: Absolutely. >> And talk about sort of some of the inside baseball. Talk about your role and how you're applying automation at VMware. >> Absolutely. So, so as part of us really running the intelligent automation program at VMware, we have a quite matured COE for last, you know four to five years, we've been doing this automation across the enterprise. So what we have really done is, you know over 45 different business functions where we really automated quite a lot different processes and tasks on that. So as part of my role, I'm really responsible for making sure that we are, you know, bringing in the best practices, making sure that we are ready to scale across the enterprise but at the same time, how, you know, quickly we are able to deliver the value of this automation to our businesses as well. >> Thomas, as a product manager, you know the product, and the market inside and out, you know the competition, you know the pricing, you know how customers are using it, you know all the features. What's your area of - main area of focus? >> The main area of the UiPathT suite... >> For your role, I mean? >> For my role is the RPA testing. So meaning testing RPA workflows themselves. And the reason is RPA has matured over the last few years. We see that, and it has adopted a lot of best practices from the software development area. So what we see is RPA now becomes business critical. It's part of the main core business processes in corporation and testing it just makes sense. You have to continuously monitor and continuously test your automation to make sure it does not break in production. >> Okay. And you have a specific product for this? Is it a feature or it's a module? >> So RPA testing or the UiPath T Suite, as the name suggests it's a suite of products. It's actually part of the existing platform. So we use Orchestrator, which is the distribution engine. We use Studio, which is our idea to create automation. And on top of that, we build a new component, which is called the UiPath Test Manager. And this is a kind of analytics and management platform where you have an oversight on what happened, what went wrong, and what is the reason for automation to **bring. >> Okay. And so Neeraj, you're testing your robot code? >> Neeraj: Correct. >> Right. And you're looking for what? Governance, security, quality, efficiency, what are the things you're looking for? >> It's actually all of all of those but our main goal to really start this was two-front, right? So we were really looking at how do we, you know, deliver at a speed with the quality which we can really maintain and sustain for a longer period, right? So to improve our quality of delivery at a speed of delivery, which we can do it. So the way we look at testing automation is not just as an independent entity. We look at this as a pipeline of a continuous improvement for us, right? So how it is called industry as a CICD pipeline. So testing automation is one of the key component of that. But the way we were able to deliver on the speed is to really have that end to end automation done for us to also from developers to production and using that pipeline and our testing is one piece of that. And the way we were able to also improve on the quality of our delivery is to really have automated way of doing the code reviews, automated way of doing the testing using this platform as well. and then, you know, how you go through end to end for that purpose. >> Thomas, when I hear testing robots, (Thomas chuckles) I don't care if it's code or actual robots, it's terrifying. >> It's terrify, yeah. >> It's terrifying. Okay, great. You, you have some test suite that says look, Yeah, we've looked at >> The, why is that terrifying? >> What's, It's terrifying because if you have to let it interact with actual live systems in some way. Yeah. The only way to know if it's going to break something is either you let it loose or you have some sort of sandbox where, I mean, what do you do? Are you taking clones of environments and running actual tests against them? I mean, think it's >> Like testing disaster recovery in the old days. Imagine. >> So we are actually not running any testing in the production live environment, right? The way we build this actually to do a testing in the separate test environment on that as well by using very specific test data from business, which you know, we call that as a golden copy of that test data because we want to use that data for months and years to come. Okay. Right? Yeah. So not touching any production environmental Facebook. >> Yeah. All right. Cause you, you can imagine >> Absolutely >> It's like, oh yeah we've created a robotic changes baby diapers let's go ahead and test it on these babies. [Collective Laughter] Yeah >> I don't think so. No, no, But, but what's the, does it does it matter if there's a delta between the test data and the, the, the production data? How, how big is that delta? How do you manage that? >> It does matter. And that's where actually that whole, you know, angle of how much you can, can in real, in real life can test right? So there are cases where you would have, even in our cases where, you know, the production data might be slightly different than the test data itself. So the whole effort goes into making sure that the test data, which we are preparing here, is as close to the products and data itself, right? It may not be a hundred percent close but that's the sort of you know, boundary or risk you may have to take. >> Okay. So you're snapshotting, that moving it over, a little V motion? >> Neeraj: Yeah. >> Okay. So do you do this for citizen developers as well? Or is you guys pretty much center of excellence writing all the bots? >> No, right now we are doing only for the unattended, the COE driven bots only at this point of time, >> What are you, what are your thoughts on the future? Because I can see I can see some really sloppy citizen coders. >> Yeah. Yeah. So as part of our governance, which we are trying to build for our citizen developers as well, there there is a really similar consideration for that as well. But for us, we have really not gone that far to build that sort of automation right >> Now, narrowly, just if we talk about testing what's the business impact been on the testing? And I'm interested in overall, but the overall platform but specifically for the testing, when did that when did you start implementing that and, and what what has been the business benefit? >> So the benefit is really on the on the speed of the delivery, which means that we are able to actually deliver more projects and more automation as well. So since we adopted that, we have seen our you know, improvement, our speed is around 15%, right? So, so, you know, 15% better speed than previously. What we have also seen is, is that our success rate of our transactions in production environment has gone to 96% success rate, which is, again there is a direct implication on business, on, on that point of view that, you know, there's no more manual exception or manual interaction is required for those failure scenarios. >> So 15% better speed at what? At, at implementing the bots? At actually writing code? Or... >> End to end, Yes. So from building the code to test that code able to approve that and then deploy that into the production environment after testing it this is really has improved by 15%. >> Okay. And, and what, what what business processes outside of sort of testing have you sort of attacked with the platform? Can you talk to that? >> The business processes outside of testing? >> Dave: Yeah. You mean the one which we are not testing ourself? >> Yeah, no. So just the UI path platform, is it exclusively for, for testing? >> This testing is exclusively for the UI path bots which we have built, right? So we have some 400 plus automations of UI bots. So it's meant exclusively >> But are you using UI path in any other ways? >> No, not at this time. >> Okay, okay. Interesting. So you started with testing? >> No, we started by building the bots. So we already had roughly 400 bots in production. When we came with the testing automation, that's when we started looking at it. >> Dave: Okay. And then now building that whole testing-- >> Dave: What are those other bots doing? Let me ask it that way. >> Oh, there's quite a lot. I mean, we have many bots. >> Dave: Paint a picture if you want. Yeah. In, in finance, in auto management, HR, legal, IT, there's a lot of automations which are there. As I'm saying, there's more than 400 automations out there. Yeah. So so it's across the, you know, enterprise on that. >> Thomas. So, and you know, both of you have a have a view on this, but Thomas's views probably wider across other, other instances. What are the most common things that are revealed in tests that indicate something needs to be fixed? Yeah, so think of, think of a test, a test failure, an error. What are the, what are the most common things that happen? >> So when we started with building our product we conducted a, a survey among our customers. And without a surprise the main reason why automation breaks is change. >> David: Sure. >> And the problem here is RPA is a controlled process a controlled workflow but it runs in an uncontrollable environment. So typically RPA is developed by a C.O.E. Those are business and automation experts, but they operate in an environment that's driven by new patches new application changes ruled out by IT. And that's the main challenge here. You cannot control that. And so far, if you, if you do not proactively test what happens is you catch an issue in production when it already breaks, right? That's reactive, that's leads to maintenance to un-claim maintenance actually. And that was the goal right from the start from the taste suite to support our customers here and go over to proactive maintenance meaning testing before and finding those issues before the heat production. >> Yeah. Yeah, yeah. So I'm, I'm still not clear on, so you just gave a perfect example, changes in the environment. >> Yeah. >> So those changes are happening in the production environment. >> Thomas: Yeah. The robot that was happily doing its automation stuff before? >> Thomas: Yeah. Everyone was happy with it. Change happens. Robot breaks. >> Thomas: Yeah. >> Okay. You're saying you test before changes are implemented? To see if those changes will break the robot? >> Thomas: Yeah. >> Okay. How do you, how do you expose those changes that are in the, in a, that are going to be in a production environment to the robot? You must have a, Is is that part of the test environment? Does that mean that you have to have what fully running instances of like an ERP system? >> Thomas: Yeah. You know, a clone of an environment. How do you, how do you test that without having the live robot against the production environment? >> I think there's no big difference to standard software testing. Okay. The interesting thing is, the change actually happens earlier. You are affected on production side with it but the change happens on it side or on DevOps side. So you typically will test in a test environment that's similar to your production environment or probably in it in a pre-product environment. And the test itself is simply running your workflow that you want to test, but mark away any dependencies you don't want to invoke. You don't want to send a, a letter to a customer in a test environment, right? And then you verify that the result is what you actually expect, right? And as soon as this is not the case, you will be notified you will have a result, the fail result, and you can act before it breaks. So you can fix it, redeploy to production and you should be good now. >> But the, the main emphasis at VMware is testing your bots, correct? >> Neeraj: Testing your bots. Yes. Can I apply this to testing other software code? >> Yeah, yeah. You, you can, you can technically actually and Thomas can speak better than me on that to any software for that matter, but we have really not explored that aspect of it. >> David: You guys have pretty good coders, good engineers at VMware, but no, seriously Thomas what's that market looking like? Is that taking off? Are you, are you are you applying this capability or customers applying it for just more broadly testing software? >> Absolutely. So our goal was we want to test RPA and the application it relies on so that includes RPA testing as well as application testing. The main difference is typical functional application testing is a black box testing. So you don't know the inner implementation of of that application. And it works out pretty well. The big, the big opportunity that we have is not isolated Not isolated testing, isolated RPA but we talk about convergence of automation. So what we offer our customers is one automation platform. You create one, you create automation, not redundantly in different departments, but you create once probably for testing and then you reuse it for RPA. So that suddenly helps your, your test engineers to to move from a pure cost center to a value center. >> How, how unique is this capability in the industry relative to your competition and and what capabilities do you have that, that or, or or differentiators from the folks that we all know you're competing with? >> So the big advantage is the power of the entire platform that we have with UiPath. So we didn't start from scratch. We have that great automation layer. We have that great distribution layer. We have all that AI capabilities that so far were used for RPA. We can reuse them, repurpose them for testing. And that really differentiates us from the competition. >> Thomas, I I, I detect a hint of an accent. Is it, is it, is it German or >> It's actually Austrian. >> Austrian. Well, >> You know. Don't compare us with Germans. >> I understand. High German. Is that the proper, is that what's spoken in Austria? >> Yes, it is. >> So, so >> Point being? >> Point being exactly as I drift off point being generally German is considered to be a very very precise language with very specific words. It's very easy to be confused about between the difference the difference between two things automation testing and automating testing. >> Thomas: Yes. >> Because in this case, what you are testing are automations. >> Thomas: Yes. >> That's what you're talking about. >> Thomas: Yes. >> You're not talking about the automation of testing. Correct? >> Well, we talk about >> And that's got to be confusing when you go to translate that into >> Dave: But isn't it both? >> 50 other languages? >> Dave: It's both. >> Is it both? >> Thomas: It actually is both. >> Okay. >> And there's something we are exploring right now which is even, even the next step, the next layer which is autonomous testing. So, so far you had an expert an automation expert creating the automation once and it would be rerun over and over again. What we are now exploring is together with university to autonomously test, meaning a bot explores your application on the test and finds issues completely autonomously. >> Dave: So autonomous testing of automation? >> It's getting more and more complicated. >> It's more clear, it's getting clearer by the minute. >> Sorry for that. >> All right Neeraj, last question is: Where do you want to take this? What's your vision for, for VMware in the context of automation? >> Sure. So, so I think the first and the foremost thing for us is to really make it more mainstream for for our automation developer Excel, right? What I mean by that is, is to really, so so there is a shift now how we engage with our business users and SMEs. And I said previously they used to actually test it manually. Now the conversation changes that, hey can you tell us what test cases you want what you want us to test in an automated measure? Can you give us the test data for that so that we can keep on testing in a continuous manner for the months and years to come down? Right? The other part of the test it changes is that, hey it used to take eight weeks for us to build but now it's going to take nine weeks because we're going to spend an extra week just to automate that as well. But it's going to help you in the long run and that's the conversation. So to really make it as much more mainstream and then say that out of all these kinds of automation and bots which we are building, So we are not looking to have a test automation for every single bot which we are building. So we need to have a way to choose where their value is. Is it the quarter end processing one? Is it the most business critical one, or is it the one where we are expecting of frequent changes, right? That's where the value of the testing is. So really bring that as a part of our whole process and then, you know >> We're still fine too. That great. Guys, thanks so much. This has been really interesting conversation. I've been waiting to talk to a real life customer about testing and automation testing. Appreciate your time. >> Thank you very much. >> Thanks for everything. >> All right. Thank you for watching, keep it right there. Dave Nicholson and I will be back right after this short break. This is day one of theCUBE coverage of UI Path Forward Five. Be right back after this short break.