(electronic music) >> Hello everybody, welcome back to Boston. This is Dave Vellante and you're watching theCUBE's continuous coverage of AWS re:Inforce 2022. We're here at the convention center in Boston where theCUBE got started in May of 2010. I'm really excited. Lena Smart is here, she's the chief information security officer at MongoDB rocket ship company We covered MongoDB World earlier this year, June, down in New York. Lena, thanks for coming to theCUBE. >> Thank you for having me. >> You're very welcome, I enjoyed your keynote yesterday. You had a big audience, I mean, this is a big deal. >> Yeah. >> This is the cloud security conference, AWS, putting its mark in the sand back in 2019. Of course, a couple of years of virtual, now back in Boston. You talked in your keynote about security, how it used to be an afterthought, used to be the responsibility of a small group of people. >> Yeah. >> You know, it used to be a bolt on. >> Yep. >> That's changed dramatically and that change has really accelerated through the pandemic. >> Yep. >> Just describe that change from your perspective. >> So when I started at MongoDB about three and a half years ago, we had a very strong security program, but it wasn't under one person. So I was their first CISO that they employed. And I brought together people who were already doing security and we employed people from outside the company as well. The person that I employed as my deputy is actually a third time returnee, I guess? So he's worked for, MongoDB be twice before, his name is Chris Sandalo, and having someone of that stature in the company is really helpful to build the security culture that I wanted. That's why I really wanted Chris to come back. He's technically brilliant, but he also knew all the people who'd been there for a while and having that person as a trusted second in command really, really helped me grow the team very quickly. I've already got a reputation as a strong female leader. He had a reputation as a strong technical leader. So us combined is like indestructible, we we're a great team. >> Is your scope of responsibility, obviously you're protecting Mongo, >> Yeah. >> How much of your role extends into the product? >> So we have a product security team that report into Sahir Azam, our chief product officer. I think you even spoke to him. >> Yeah, he's amazing. >> He's awesome, isn't he? He's just fabulous. And so his team, they've got security experts on our product side who are really kind of the customer facing. I'm also to a certain extent customer facing, but the product folks are the absolute experts. They will listen to what our customers need, what they want, and together we can then work out and translate that. I'm also responsible for governance risk and compliance. So there's a large portion of our customers that give us input via that program too. So there's a lot of avenues to allow us to facilitate change in the security field. And I think that's really important. We have to listen to what our customers want, but also internally. You know, what our internal groups need as well to help them grow. >> I remember last year, Re:invent 2021, I was watching a talk on security. It was the, I forget his name, but it was the individual who responsible for data center security. And one of the things he said was, you know, look it's not at the end of the day, the technology's important but it's not the technology. It's how you apply the tools and the practices and the culture- >> Right. That you build in the organization that will ultimately determine how successful you are at decreasing the ROI for the bad guys. >> Yes. >> Let's put it that way. So talk about the challenges of building that culture, how you go about that, and how you sustain that cultural aspect. >> So, I think having the security champion program, so that's just, it's like one of my babies, that and helping underrepresented groups in MongoDB kind of get on in the tech world are both really important to me. And so the security champion program is purely voluntary. We have over a hundred members. And these are people, there's no bar to join. You don't have to be technical. If you're an executive assistant who wants to learn more about security, like my assistant does, you're more than welcome. Up to, we actually people grade themselves, when they join us, we give them a little tick box. Like five is, I walk in security water. One is, I can spell security but I'd like to learn more. Mixing those groups together has been game changing for us. We now have over a hundred people who volunteer their time, with their supervisors permission, they help us with their phishing campaigns, testing AWS tool sets, testing things like queryable encryption. I mean, we have people who have such an in-depth knowledge in other areas of the business that I could never learn, no matter how much time I had. And so to have them- And we have people from product as security champions as well, and security, and legal, and HR, and every department is recognized. And I think almost every geographical location is also recognized. So just to have that scope and depth of people with long tenure in the company, technically brilliant, really want to understand how they can apply the cultural values that we live with each day to make our security program stronger. As I say, that's been a game changer for us. We use it as a feeder program. So we've had five people transfer from other departments into the security and GRC teams through this Champions program. >> Makes a lot of sense. You take somebody who walks on water in security, mix them with somebody who really doesn't know a lot about it but wants to learn and then can ask really basic questions, and then the experts can actually understand better how to communicate. >> Absolutely. >> To that you know that 101 level. >> It's absolutely true. Like my mom lives in her iPad. She worships her iPad. Unfortunately she thinks everything on it is true. And so for me to try and dumb it down, and she's not a dumb person, but for me to try and dumb down the message of most of it's rubbish, mom, Facebook is made up. It's just people telling stories. For me to try and get that over to- So she's a one, and I might be a five, that's hard. That's really hard. And so that's what we're doing in the office as well. It's like, if you can explain to my mother how not everything on the internet is true, we're golden. >> My mom, rest her soul, when she first got a- we got her a Macintosh, this was years and years and years ago, and we were trying to train her over the phone, and said, mom, just grab the mouse. And she's like, I don't like mice. (Lena laughs) There you go. I know, I know, Lena, what that's like. Years ago, it was early last decade, we started to think about, wow, security really has to become a board level item. >> Yeah. >> And it really wasn't- 2010, you know, for certain companies. But really, and so I had the pleasure of interviewing Dr. Robert Gates, who was the defense secretary. >> Yes. >> We had this conversation, and he sits on a number, or sat on a number of boards, probably still does, but he was adamant. Oh, absolutely. Here's how you know, here. This is the criticality. Now it's totally changed. >> Right. >> I mean, it's now a board level item. But how do you communicate to the C-Suite, the board? How often do you do that? What do you recommend is the right regime? And I know there's not any perfect- there's got to be situational, but how do you approach it? >> So I am extremely lucky. We have a very technical board. Our chairman of the board is Tom Killalea. You know, Amazon alum, I mean, just genius. And he, and the rest of the board, it's not like a normal board. Like I actually have the meeting on this coming Monday. So this weekend will be me reading as much stuff as I possibly can, trying to work out what questions they're going to ask me. And it's never a gotcha kind of thing. I've been at board meetings before where you almost feel personally attacked and that's not a good thing. Where, at MongoDB, you can see they genuinely want us to grow and mature. And so I actually meet with our board four times a year, just for security. So we set up our own security meeting just with board members who are specifically interested in security, which is all of them. And so this is actually off cadence. So I actually get their attention for at least an hour once a quarter, which is almost unheard of. And we actually use the AWS memo format. People have a chance to comment and read prior to the meeting. So they know what we're going to talk about and we know what their concerns are. And so you're not going in like, oh my gosh, what what's going to happen for this hour? We come prepared. We have statistics. We can show them where we're growing. We can show them where we need more growth and maturity. And I think having that level of just development of programs, but also the ear of the board has has helped me mature my role 10 times. And then also we have the chance to ask them, well what are your other CISOs doing? You know, they're members of other boards. So I can say to Dave, for example, you know, what's so-and-so doing at Datadog? Or Tom Killelea, what's the CISO of Capital One doing? And they help me make a lot of those connections as well. I mean, the CISO world is small and me being a female in the world with a Scottish accent, I'm probably more memorable than most. So it's like, oh yeah, that's the Irish girl. Yeah. She's Scottish, thank you. But they remember me and I can use that. And so just having all those mentors from the board level down, and obviously Dev is a huge, huge fan of security and GRC. It's no longer that box ticking exercise that I used to feel security was, you know, if you heated your SOC2 type two in FinTech, oh, you were good to go. You know, if you did a HERC set for the power industry. All right, right. You know, we can move on now. It's not that anymore. >> Right. It's every single day. >> Yeah. Of course. Dev is Dev at the Chario. Dev spelled D E V. I spell Dave differently. My Dave. But, Lena, it sounds like you present a combination of metrics, so, the board, you feel like that's appropriate to dig into the metrics. But also I'm presuming you're talking strategy, potentially, you know, gaps- >> Road roadmaps, the whole nine yards. Yep. >> What's the, you know, I look at the budget scenario. At the macro level, CIOs have told us, they came into the year saying, hey we're going to grow spending at the macro, around eight percent, eight and a half percent. That's dialed down a little bit post Ukraine and the whole recession and Fed tightening. So now they're down maybe around six percent. So not dramatically lower, but still. And they tell us security is still the number one priority. >> Yes. >> That's been the case for many, many quarters, and actually years, but you don't have an unlimited budget. >> Sure >> Right. It's not like, oh, here is an open checkbook. >> Right. >> Lena, so, how does Mongo balance that with the other priorities in the organization, obviously, you know, you got to spend money on product, you got to spend money and go to market. What's the climate like now, is it, you know continuing on in 2022 despite some of the macro concerns? Is it maybe tapping the brakes? What's the general sentiment? >> We would never tap the breaks. I mean, this is something that's- So my other half works in the finance industry still. So we have, you know, interesting discussions when it comes to geopolitics and financial politics and you know, Dev, the chairman of the board, all very technical people, get that security is going to be taken advantage of if we're seeing to be tapping the brakes. So it does kind of worry me when I hear other people are saying, oh, we're, you know, we're cutting back our budget. We are not. That being said, you also have to be fiscally responsible. I'm Scottish, we're cheap, really frugal with money. And so I always tell my team: treat this money as if it's your own. As if it's my money. And so when we're buying tool sets, I want to make sure that I'm talking to the CISO, or the CISO of the company that's supplying it, and saying are you giving me the really the best value? You know, how can we maybe even partner with you as a database platform? How could we partner with you, X company, to, you know, maybe we'll give you credits on our platform. If you look to moving to us and then we could have a partnership, and I mean, that's how some of this stuff builds, and so I've been pretty good at doing that. I enjoy doing that. But then also just in terms of being fiscally responsible, yeah, I get it. There's CISOs who have every tool that's out there because it's shiny and it's new and they know the board is never going to say no, but at some point, people will get wise to that and be like, I think we need a new CISO. So it's not like we're going to stop spending it. So we're going to get someone who actually knows how to budget and get us what the best value for money. And so that's always been my view is we're always going to be financed. We're always going to be financed well. But I need to keep showing that value for money. And we do that every board meeting, every Monday when I meet with my boss. I mean, I report to the CFO but I've got a dotted line to the CTO. So I'm, you know, I'm one of the few people at this level that's got my feet in both camps. You know budgets are talked at Dev's level. So, you know, it's really important that we get the spend right. >> And that value is essentially, as I was kind of alluding to before, it's decreasing the value equation for the hackers, for the adversary. >> Hopefully, yes. >> Right? Who's the- of course they're increasingly sophisticated. I want to ask you about your relationship with AWS in this context. It feels like, when I look around here, I think back to 2019, there was a lot of talk about the shared responsibility model. >> Yes. >> You know, AWS likes to educate people and back then it was like, okay, hey, by the way, you know you got to, you know, configure the S3 bucket properly. And then, oh, by the way, there's more than just, it's not just binary. >> Right, right. >> There's other factors involved. The application access and identity and things like that, et cetera, et cetera. So that was all kind of cool. But I feel like the cloud is becoming the first line of defense for the CISO but because of the shared responsibility model, CISO is now the second line of defense >> Yes. Does that change your role? Does it make it less complicated in a way? Maybe, you know, more complicated because you now got to get your DevSecOps team? The developers are now much more involved in security? How is that shifting, specifically in the context of your relationship with AWS? >> It's honestly not been that much of a shift. I mean, these guys are very proactive when it comes to where we are from the security standpoint. They listen to their customers as much as we do. So when we sit down with them, when I meet with Steve Schmidt or CJ or you know, our account manager, its not a conversation that's a surprise to me when I tell them this is what we need. They're like, yep, we're on that already. And so I think that relationship has been very proactive rather than reactive. And then in terms of MongoDB, as a tech company, security is always at the forefront. So it's not been a huge lift for me. It's really just been my time that I've taken to understand where DevSecOps is coming from. And you know, how far are we shifting left? Are we actually shifting right now? It's like, you know, get the balance, right? You can't be too much to one side. But I think in terms of where we're teaching the developers, you know, we are a company by developers for developers. So, we get it, we understand where they're coming from, and we try and be as proactive as AWS is. >> When you obviously the SolarWinds hack was a a major mile- I think in security, there's always something in the headlines- >> Yes. But when you think of things like, you know, Stuxnet, you know, Log4J, obviously Solarwinds and the whole supply chain infiltration and the bill of materials. As I said before, the adversary is extremely capable and sophisticated and you know, much more automated. It's always been automated attacks, but you know island hopping and infiltrating and self-forming malware and really sophisticated techniques. >> Yep. >> How are you thinking about that supply chain, bill of materials from inside Mongo and ultimately externally to your customers? >> So you've picked on my third favorite topic to talk about. So I came from the power industry before, so I've got a lot of experience with critical infrastructure. And that was really, I think, where a lot of the supply chain management rules and regulations came from. If you're building a turbine and the steel's coming from China, we would send people to China to make sure that the steel we were buying was the steel we were using. And so that became the H bomb. The hardware bill of materials, bad name. But, you know, we remember what it stood for. And then fast forward: President Biden's executive order. SBOs front and center, cloud first front and center. It's like, this is perfect. And so I was actually- I actually moderated a panel earlier this year at Homeland Security Week in DC, where we had a sneak CISA, So Dr. Allen Friedman from CISA, and also Patrick Weir from OWASP for the framework, CISA for the framework as well, and just the general guidance, and Snake for the front end. That was where my head was going. And MongoDB is the back-end database. And what we've done is we've taken our work with Snake and we now have a proof of concept for SBOs. And so I'm now trying to kind of package that, if you like, as a program and get the word out that SBOs shouldn't be something to be afraid of. If you want to do business with the government you're going to have to create one. We are offering a secure repository to store that data, the government could have access to that repository and see that data. So there's one source of truth. And so I think SBOs is going to be really interesting. I know that, you know, some of my peers are like, oh, it's just another box to tick. And I think it's more than that. I definitely- I've just, there's something percolating in the back of my mind that this is going to be big and we're going to be able to use it to hopefully not stop things like another Log4j, there's always going to be another Log4j, we know that. we don't know everything, the unknown unknown, but at least if we're prepared to go find stuff quicker than we were then before Log4j, I think having SBOs on hand, having that one source of truth, that one repository, I think is going to make it so much easier to find those things. >> Last question, what's the CISO's number one challenge? Either yours or the CISO, generally. >> Keeping up with the fire hose that is security. Like, what do you pick tomorrow? And if you pick the wrong thing, what's the impact? So that's why I'm always networking and talking to my peers. And, you know, we're sometimes like meerkats, you know. there's meerkats, you see like this, it's like, what do we talk about? But there's always something to talk about. And you just have to learn and keep learning. >> Last question, part B. As a hot technology company, that's, you know, rising star, you know not withstanding the tech lash and the stock market- >> Yeah. >> But Mongo's growing, you know, wonderfully. Do you find it easier to attract talent? Like many CISOs will say, you know, lack of talent is my biggest, biggest challenge. Do you find that that's not the challenge for you? >> Not at all. I think on two fronts, one, we have the champions program. So we've got a whole internal ecosystem who love working there. So the minute one of my jobs goes on the board, they get first dibs at it. So they'd already phoning their friends. So we've got, you know, there's ripple effects out from over a hundred people internally. You know, I think just having that, that's been a game changer. >> I was so looking forward to interviewing you, Lena, thanks so much for coming. >> Thank you, this was a pleasure. >> It was really great to have you. >> Thank you so much. Thank you. >> You're really welcome. All right, keep it right there. This is Dave Villante for theCUBE. We'll be right back at AWS Re:inforce22 right after this short break.

hi everybody welcome back to boston you're watching thecube's coverage of reinforce 2022 last time we were here live was 2019. had a couple years of virtual merit bear is here she's with the office of the cso for aws merit welcome back to the cube good to see you thank you for coming on thank you so much it's good to be back um yes cso chief information security officer for folks who are acronym phobia phobic yeah okay so what do you do for the office of the is it ciso or sizzo anyway ah whatever is it sim or theme um i i work in three areas so i sit in aws security and i help us do security we're a shop that runs on aws i empathize with folks who are running shops it is process driven it takes hard work but we believe in certain mechanisms and muscle groups so you know i work on getting those better everything from how we do threat intelligence to how we guard rail employees and think about vending accounts and those kinds of things i also work in customer-facing interactions so when a cso wants to meet awssc so that's often me and then the third is product side so ensuring that everything we deliver not just security services are aligned with security best practices and expectations for our customers so i have to ask you right off the bat so we do a lot of spending surveys we have a partner etr i look at the data all the time and for some reason aws never shows up in the spending metrics why do you think that is maybe that talks to your strategy let's double click on that yeah so first of all um turn on guard duty get shield advanced for the you know accounts you need the 3k is relatively small and a large enterprise event like this doesn't mean don't spend on security there is a lot of goodness that we have to offer in ess external security services but i think one of the unique parts of aws is that we don't believe that security is something you should buy it's something that you get from us it's something that we do for you a lot of the time i mean this is the definition of the shared responsibility model right everything that you interact with on aws has been subject to the same rigorous standards and we aws security have umbrella arms around those but we also ensure that service teams own the security of their service so a lot of times when i'm talking to csos and i say security teams or sorry service teams own the security of their service they're curious like how do they not get frustrated and the answer is we put in a lot of mechanisms to allow those to go through so there's automation there are robots that resolve those trouble tickets you know like and we have emissaries we call them guardian champions that are embedded in service teams at any rate the point is i think it's really beautiful the way that customers who are you know enabling services in general benefit from the inheritances that they get and in some definition this is like the value proposition of cloud when we take care of those lower layers of the stack we're doing everything from the concrete floors guards and gates hvac you know in the case of something like aws bracket which is our quantum computing like we're talking about you know near vacuum uh environments like these are sometimes really intricate and beautiful ways that we take care of stuff that was otherwise manual and ugly and then we get up and we get really intricate there too so i gave a talk this morning about ddos protection um and all the stuff that we're doing where we can see because of our vantage point the volume and that leads us to be a leader in volumetric attack signatures for example manage rule sets like that costs you nothing turn on your dns firewall like there are ways that you just as a as an aws customer you inherit our rigorous standards and you also are able to benefit from the rigor with which we you know exact ourselves to really you're not trying to make it a huge business at least as part of your your portfolio it's just it's embedded it's there take advantage of it i want everyone to be secure and i will go to bad to say like i want you to do it and if money is a blocker let's talk about that because honestly we just want to do the right thing by customers and i want customers to use more of our services i genuinely believe that they are enablers we have pharma companies um that have helped enable you know personalized medicine and some of the copic vaccines we have you know like there are ways that this has mattered to people in really intimate ways um and then fun ways like formula one uh you know like there are things that allow us to do more and our customers to do more and security should be a way of life it's a way of breathing you don't wake up and decide that you're going to bolt it on one day okay so we heard cj moses keynote this morning i presume you were listening in uh we heard a lot about you know cool tools you know threat detection and devops and container security but he did explicitly talked about how aws is simplifying the life of the cso so what are you doing in that regard and what's that that's let's just leave it there for now i talk to c sales every day and i think um most of them have two main concerns one is how to get their organization to grow up like to understand what security looks like in a cloudy way um and that means that you know your login monitoring is going to be the forensics it's not going to be getting into the host that's on our side right and that's a luxury like i think there are elements of the cso job that have changed but that even if you know cj didn't explicitly call them out these are beauties things like um least privilege that you can accomplish using access analyzer and all these ways that inspector for example does network reachability and then all of these get piped to security hub and there's just ways that make it more accessible than ever to be a cso and to enable and embolden your people the second side is how csos are thinking about changing their organization so what are you reporting to the board um how are you thinking about hiring and um in the metrics side i would say you know being and i get a a lot of questions that are like how do we exhibit a culture of security and my answer is you do it you just start doing it like you make it so that your vps have to answer trouble tickets you may and and i don't mean literally like every trouble ticket but i mean they are 100 executives will say that they care about security but so what like you know set up your organization to be responsive to security and to um have to answer to them because it matters and and notice that because a non-decision is a decision and the other side is workforce right and i think um i see a lot of promise some of it unfulfilled in folks being hired to look different than traditional security folks and act different and maybe a first grade teacher or an architect or an artist and who don't consider themselves like particularly technical like the gorgeousness of cloud is that you can one teach yourself this i mean i didn't go to school for computer science like this is the kind of thing we all have to teach ourselves but also you can abstract on top of stuff so you're not writing code every day necessarily although if you are that's awesome and we love debbie folks but you know there's there's a lot of ways in which the machine of the security organization is suggesting i think cj was part to answer your question pointedly i think cj was trying to be really responsive to like all the stuff we're giving you all the goodness all the sprinkles on your cupcake not at all the organizational stuff that is kind of like you know the good stuff that we know we need to get into so i think so you're saying it's it's inherent it's inherently helping the cso uh her life his life become less complex and i feel like the cloud you said the customers are trying to become make their security more cloudy so i feel like the cloud has become the first line of defense now the cso your customer see so is the second line of defense maybe the audit is the third line what does that mean for the role of the the cso how is that they become a compliance officer what does that mean no no i think actually increasingly they are married or marriable so um when you're doing so for example if you are embracing [Music] ephemeral and immutable infrastructure then we're talking about using something like cloud formation or terraform to vend environments and you know being able to um use control tower and aws organizations to dictate um truisms through your environment you know like there are ways that you are basically in golden armies and you can come back to a known good state you can embrace that kind of cloudiness that allows you to get good to refine it to kill it and spin up a new infrastructure and that means though that like your i.t and your security will be woven in in a really um lovely way but in a way that contradicts certain like existing structures and i think one of the beauties is that your compliance can then wake up with it right your audit manager and your you know security hub and other folks that do compliance as code so you know inspector for example has a tooling that can without sending a single packet over the network do network reachability so they can tell whether you have an internet facing endpoint well that's a pci standard you know but that's also a security truism you shouldn't have internet facing endpoints you don't approve up you know like so these are i think these can go in hand in hand there are certainly i i don't know that i totally disregard like a defense in-depth notion but i don't think that it's linear in that way i think it's like circular that we hope that these mechanisms work together that we also know that they should speak to each other and and be augmented and aware of one another so an example of this would be that we don't just do perimeter detection we do identity-based fine-grained controls and that those are listening to and reasoned about using tooling that we can do using security yeah we heard a lot about reasoning as well in the keynote but i want to ask about zero trust like aws i think resisted using that term you know the industry was a buzzword before the pandemic it's probably more buzzy now although in a way it's a mandate um depending on how you look at it so i mean you anything that's not explicitly allowed is denied in your world and you have tools and i mean that's a definition if it's a die that overrides if it's another it's a deny call that will override and allow yeah that's true although anyway finish your question yeah yeah so so my it's like if there's if there's doubt there's no doubt it seems in your world but but but you have a lot of capabilities seems to me that this is how you you apply aws internal security and bring that to your customers do customers talk to you about zero trust are they trying to implement zero trust what's the best way for them to do that when they don't have that they have a lack of talent they don't have the skill sets uh that it and the knowledge that aws has what are you hearing from customers in that regard yeah that's a really um nuanced phrasing which i appreciate because i think so i think you're right zero trust is a term that like means everything and nothing i mean like this this notebook is zero trust like no internet comes in or out of it like congratulations you also can't do business on it right um i do a lot of business online you know what i mean like you can't uh transact something to other folks and if i lose it i'm screwed yeah exactly i usually have a water bottle or something that's even more inanimate than your notebook um but i guess my point is we i don't think that the term zero trust is a truism i think it's a conceptual framework right and the idea is that we want to make it so that someone's position in the network is agnostic to their permissioning so whereas in the olden days like a decade ago um we might have assumed that when you're in the perimeter you just accept everything um that's no longer the right way to think about it and frankly like covid and work from home may have accelerated this but this was ripe to be accelerated anyway um what we are thinking about is both like you said under the network so like the network layer are we talking about machine to machine are we talking about like um you know every api call goes over the open internet with no inherent assurances human to app or it's protected by sig v4 you know like there is an inherent zero trust case that we have always built this goes back to a jeff bezos mandate from 2002 that everything be an api call that is again this kind of like building security into it when we say security is job zero it not only reflects the fact that like when you build a terraform or a cloud formation template you better have permission things appropriately or try to but also that like there is no cloud without security considerations you don't get to just bolt something on after the fact so that being said now that we embrace that and we can reason about it and we can use tools like access analyzer you know we're also talking about zero trust in that like i said augmentation identity centric fine grained controls so an example of this would be a vpc endpoint policy where it is a perm the perimeter is dead long live the perimeter right you'll have your traditional perimeter your vpc or your vpn um augmented by and aware of the fine-grained identity-centric ones which you can also reason about prune down continuously monitor and so on and that'll also help you with your logging and monitoring because you know what your ingress and egress points are how concerned should people be with quantum messing up all the encryption algos oh it's stopping created right okay so but we heard about this in the keynote right so is it just a quantum so far off by the time we get there is it like a y2k you're probably not old enough to remember y2k but y2k moment right i mean i can't take you anywhere what should we um how should we be thinking about quantum in the context of security and sure yeah i mean i think we should be thinking about quantum and a lot of dimensions as operationally interesting and how we can leverage i think we should be thinking about it in the security future for right now aes256 is something that is not broken so we shouldn't try to fix it yeah cool encrypt all the things you can do it natively you know like i love talking about quantum but it's more of an aspirational and also like we can be doing high power compute to solve problems you know but like for it to get to a security uh potentially uh vulnerable state or like something that we should worry about is a bit off yeah and show me an application that can yeah and i mean and i think at that point we're talking about homomorphic improvements about another thing i kind of feel the same way is that you know there's a lot of hype around it a lot of ibm talks about a lot you guys talked about in your keynote today and when i really talk to people who understand this stuff it seems like it's a long long way off i don't think it's a long long way off but everything is dog years in tech world but um but for today you know like for today encrypt yourself we will always keep our encryption up to standard and you know that will be for now like the the industry grade standard that folks i mean like i i have i have never heard of a case where someone had their kms keys broken into i um i always ask like awesome security people this question did you like how did you get into this did you have like did you have a favorite superhero as a kid that was going to save the world i um was always the kid who probably would have picked up a book about the cia and i like find this and i don't remember who i was before i was a security person um but i also think that as a woman um from an american indian family walking through the world i think about the relationship between dynamics with the government and companies and individuals and how we want to construct those and the need for voices that are observant of the ways that those interplay and i always saw this as a field where we can do a lot of good yeah amazing merritt thanks so much for coming on thecube great guest john said you would be really appreciate your time of course all right keep it ready you're very welcome keep it right there this is dave vellante for the cube we'll be right back at aws reinforced 2022 from boston keep right there [Music]

>> From Gillette Stadium, in Foxborough, Massachusetts, it's theCUBE covering VTUG Winter Warmer 2019, brought to you by SiliconANGLE Media. >> Hi I'm Stu Miniman and this is theCUBE at VTUG Winter Warmer 2019 at Gillette Stadium, home of the New England Patriots, AFC Champions week out from going to Super Bowl 53. Joining me is a user from the great state of Maine, Dawnna Pease, who is the Director of Computing Infrastructure and Services for the state of Maine, thank you for joining us. >> Yes, thank you. >> Alright so Dawnna, you've been to a few VTUGs, of course the Summer Fest, which is, you know it might not be quite as big, as the winter one, but it is known even broader, I've known people come from out of the country because there's a giant lobster bake at the end of the day. I've been a few times, but you know tell us, you've been to VTUG before, yes? >> We have, so I have been to many, especially in Maine. And this is probably our fourth or fifth one that I've broughten the team from the state of Maine here and I feel it's really crucial and important because it allows them to network, to talk with their peers and to look at the technologies of how we can provide services for the constituents of the state of Maine and for our services that we offer within our office. >> Yeah so we always love talking to the users, we love to be able to help you share with your peers what you've been learning and actually I've had lots of great government discussions over the last few years, even attended, I attended a public sector show in the cloud space last year, and it's always fascinating because people have a misconception when it comes to what it's like to be IT in government, so let's dig into that a little bit. Tell us a little bit about your role, your group, what's kind of under your purview. >> Sure, I've been in state government going on 33 years as a public servant, very proud of that. I have a great group and I am the Director of Computing Infrastructure Services and it's really directory services, Microsoft stack. We have VMWare environment that we been probably nine years now and we're just implementing SimpliVity our hyperconverged, and after extensive research on that, we really solidified and selected HPE SimpliVity because in state government we had a lot of aging servers that needed to be replaced as well as our VM environment which was 44 nodes and it was a huge investment so not only on the licensing, hardware, storage, the compute part as well. So lookin' at the hyperconverged that was just one of many of our technologies that we looked at. >> So Dawnna take us back, how long ago did you start looking at that initiative? >> Oh 18 months. >> Okay, and was it a single location, multiple locations, can you give us any, how many you know servers or VMs or locations that this solution was going to span? >> For me it was actually spannin' and takin' on many of our on-prem solutions that we have. Like our SQL environment, our application hostin', the one offs, we're bringin' into that. As well as upgrading our existing VM cluster. So it's really taken on and morphed even more. We have a lot of net new as that want to participate in this environment so for us it is literally like a cloud solution, but it's for within our own private cloud solution on that. >> And these were critical business productivity applications that you're talking about? >> Absolutely >> This wasn't a new project to do, you know, early days of hyper converged, it was like oh I'm doing desktop virtualization, let me roll this out. I mean you're talking about databases and applications. >> Absolutely so we run close to, little over 600 servers for virtual and physical, so when all said and done within our hyperconverged our goal is to really be under 60 physicals left within state government. And currently today we have probably over 400 in our virtual environment today. So we're really expanding that more and bringing the services all into one knowing that we're going to have compute network and everything in our storage will all be in this environment. Plus we have a legacy storage environment, so when you're thinking of your legacy storage environment and you're looking at your refreshment of hardware and all the licenses around that our return on investment was huge for the state of Maine. So it was literally the wise choice for us to do within state government for tax payers, saving money. Also for the state as a whole. >> I have to imagine in addition to kind of the Capex piece if you're saying going from 900 to 400 and looking to get down to 60, operationally hopefully it makes the jobs of you know you and your team, a little bit easier once things are up and running. And that's one of the promises of hyperconverged, is it should be that cloud layer, it should be almost invisible when you talk about, it's just a pool that my virtualization lives on but I don't need to touch and rack and stack stuff the way that I might have in the past. >> Exactly, exactly, good point on that. Also on that we've really taken a broad look at how we can leverage the cloud so from a disaster recovery aspect and not only havin' the site resilience between two data centers, but how we can leverage the cloud for that continuity aspect. So we're really broadening that and the team's doing a fabulous, excellent job at that. >> Are you doing the Cloud DR today or is that a future plan? >> That is future. >> Okay, going to leverage a public cloud as that Are you far enough down? >> Government. So we have Azure today and we have a government tenant on that so we will use that aspect within the government tenant as well. >> Great so primarily Microsoft applications, you've moved into hyperconverged and you leveraged the Azure government certified cloud pieces. >> Correct >> Okay, awesome, when you started going down this path did you have in your mind hyperconverged or is that, how did you end up on that type of solution? >> So no, we didn't. Doin' the research on that and lookin' at all options, and really doin' the research with that, hyperconverged was more of makin' sense from the return on investment and also from a ... I want to say the simplified fashion, like you said it's simple you want to make it not so complex, it provided everything within that environment, and it was really based on how we were structured today, the investment that we would need to do if didn't go down this path. And taking in, so we did go with the hyperconverged. >> In your previous environment were you using HPE for the servers or the storage? >> So we were HPE, we are an HPE shop. And we have VMC, we have Pure Storage, we have different aspects of our storage today that exist so lookin' at that as well, we had an investment that we either needed to upgrade, replace, and, or invest. >> What I was poking at a little bit is were you HPE before, was that part of the decision to buy SimpliVity which is part of the HPE family or was that not a major factor? >> It was not a major factor, I mean we were ... We have always been a HPE shop, however we had criteria we were lookin' at, so you know after doing the research and we had 15, we were lookin' at 15 vendors at the time. We narrowed it down to like eight, and out of that we really narrowed it down to two that were in the quadrant, in the Gartner quadrant. And in doing our own research and study and bringin' all the vendors in and everything and what we had already invested what we currently had, it really came out to SimpliVity as the choice. >> And your 18 months into this, you've got some Cloud DR in the future, how are things going? What have you learned so far, is there anything you would have done differently or any advice you'd give to your peers if they're starting to go down this path? >> Do the research, do the research, be very thorough in what you're lookin' at for your requirements. And you know not only the research but look at what you've already invested in and take that into consideration and what your return on investment, what you're looking for your return on investment because you need to look just past not only your hosting environment but it really goes into can your network support that environment? Do you need to upgrade your network, your storage aspects, licensing aspects of that as well? So it's a huge investment, however look at the money they already pay in. >> Yeah licensing, one of those things when you talk about that great reduction of servers, are you today or do you expect in the future some of those licensing costs from the database, the virtualization, will those actually be able to be scaled down? >> Absolutely, and that was part of our ROI as well. By a lot, you know and that is one of the benefits of the hyperconverged as well. Once you set that up and purchase the proper licenses, I mean like data center licenses, you can put in as many VMs as you need within that environment and that's important. So you're really just looking at your compute at that, what you need for storage and compute. >> Yeah, I'm curious just spoke, cause we have, we've worked with clients for years on that and often times I've got a ELA or I've got a multi-year contract there and I have to renegotiate it, has that gone smoothly? Have there been any bumps along the road or is it pretty straightforward that licensing can be a huge chunk of your budget and like oh great, I'm two years later and I'm going to save myself a lot of money. >> So I actually am the administrator of our enterprise agreement with Microsoft, had been for many years, so I know what we have. And so I work very closely with that and I as far as the licensing and what we have, so for the renewals, I will say it gets easier. I found that being consolidated because when the agencies own their IT, at the time, we had many enterprise agreements and that was more complex so if you can actually consolidate and go into one, we have one enterprise agreement, or under the three I would say, it's much more manageable on that. So I don't find that that's a show stopper on that, it's gotten easier over the years. Simplified, it's more simplified. >> It's great to hear that and actually Microsoft has made great strides, Microsoft today is not the Microsoft of fives years ago or 10 years ago. >> Correct, I would agree. >> So, Dawnna Pease, pleasure talking to you. Thank you so much for sharing your experiences and be sure to check out thecube.net for all the recordings from the VTUG Winter Warmer 2019 as well as all of the other shows. I'm Stu Miniman and thank you for watching theCUBE.

>> Live from Orlando, Florida. It's theCUBE, covering Cisco Live 2018. Brought to you by Cisco, NetApp, and theCUBE's ecosystem partners. >> Okay welcome back everyone, we're here live in the Cisco DevNet Zone, at Cisco Live 2018. It's theCUBE's exclusive coverage. This is Go Live, I'm John Furrier with Stu Miniman there, here with Suzie Wee who is the CTO and Vice President of Cisco. This is her baby DevNet, the fastest growing developer program in Cisco history, only four years old. Welcome to theCUBE, good to see you again. >> Hey John good to see you, hey Stu. >> I made that stat, it was only four years old. So DevNet, obviously just for color commentary, really successful developer program, only in it's fourth year or so for Cisco. But it's really changing the face of Cisco. It's showing that a new collaboration, a new co-development, a new developer framework is being built on top of networks and it's on a collision course with Cloud Native. Kay, this is a great path for network engineers. It really changed the show vibe so congratulations. >> Thank you, thank you. Yeah, and why do you say collision course? There's like a whole new paradigm, right? And it's pretty amazing, it's pretty amazing. >> Well some of the things that we've been seeing here, obviously CCIE's or 25 years of excellence and stats was out here >> Yes, Yes. >> The key note from the CEO, Chuck Robbins, talks about an old way and new way. Developers are clearly in the driver's seat here and network engineers, Cisco partners, customers technical folks and engineers. They're at the keys to the kingdom and you introduced a concept called Network Dev Ops. >> Yes. >> Okay, a few years ago when we first had you on theCUBE. Where is that now? Where is Network Dev Ops now? What's the vibe internally? Is there a full acceptance to it? Is there embracing it? >> It's amazing and ya know it's like, when we were pushing it we were just saying, "Hey, the network is changing, the network "is gonna be programmable, the network "is going to have API's", and you go back four years and then you're just like, "What was the buzz?" The buzz was SDN, y'know the buzz was SDN. SDN was open flow, it was separation of control plain from data plain. But, it was still kind of research. And what we knew is like, it wouldn't become real until the people who are building and operating the World's networks were ready to adopt it. And so, at first of course, it was like, there were the people who were like, "Okay this network thing, this programmability "is gonna come to the network, but what can we do there?" And since then, people have jumped in, they've like really gotten in. And like here at this Cisco Live, what we're seeing is that people are ready to code. And so the concept of, I'm a networker, now there's software built into my entire network programming portfolio. How do I build the skills? I'm a developer, and the networkers are getting comfortable with understanding that they need to code, they need to understand these skills. But one thing that we did, was we actually separated out, like, the definition of developer. >> Yep. >> Y'know. >> You guys done a good job of really defining a path for the network engineer, who can extend their skill set and solve network problems, be creative, and also do great business outcome oriented things. So, I want you to take a minute to explain the DevNet story because you guys just didn't throw a PowerPoint at this. You dug in, you built it up, and you threw a lot of resources for Cisco, I mean small for Cisco's scale, but you guys dug in, you did the homework and you're doing new things. So take us to the DevNet story and what's happening this year in the momentum. Take us through that little journey. >> Yeah, so the story was back in actually 2013. Cisco was saying, "Hey, we're gonna get into software "we're doing software, we have a software strategy." And all of that is fantastic, either... But the thing that was missing, was like, Hey, we need an ecosystem, like the reason you do software is to have an ecosystem. And in order to have an ecosystem you want people to build upon your stuff. You need to expose your API's. It doesn't happen by itself, you need to have a developer program so that you can actually really let people use all of that and partake in the ecosystem. So we, kind of, I evangelized, evangelized, evangelized, gave a couple hundred pitches, got the okay to start DevNet, and that was in 2014. And then in 2014, then we said okay. So now we got the okay to start a developer program for Cisco. But, y'know, it's still not a sure shot that it would work. >> Yeah. >> And then we said our dream is to have a developer conference at Cisco Live. And so we wanted to have that developer conference at Cisco Live and then three months later, we had it. And we're like okay, 24 hour hack-a-thon, deep dive API sessions, but would the people come? Would they be ready? And then, they came. Like, they came, it was packed. It was just like wall to wall of people, who are excited to learn about software. So now you go and then you fast forward, y'know, four years, and now we just hit 500,000 developers. 500,000 people have registered for DevNet. And you can be like, "Well what does that mean?" We have half a million developers. Is it a real number? Well, my team kept scrubbing the database. Like so, we had hit 400,000 and then our numbers got lower and I was like "Come on guys, stop it!" And they were like, "No, no, no, we have to scrub it, "we gotta out the duplicates." And then finally we got it up and we've grown it. It basically is at 500,000 registered developers. And what that means is like, now we have a community. We have a community of people who are getting up on network API's, we have a community of people who can develop, and once you do that you hit this completely different inflection point. Where at first our mission was just to help networkers be developers, to help the app developers understand that the network has API's and to do stuff there. That's still our goal, to enable developers. But now we have a community, what we can do is really catalyze that community into business and impact. >> Suzie, first of all congratulations. It's been so much fun to be here in the DevNet Zone. It'd been a few years since I'd been to Cisco Live. And y'know, people in these sessions every time. And you go, people are coding, they're white-boarding, they're, y'know building. Playing with Legos, they're doing all sorts of stuff. Over the last five years, y'know, we all knew that, y'know, developers of the new Kingmakers. It's been talked about a lot. But we've seen many infrastructure companies try. They create little developer conferences, they bring in speakers, they'll get some momentum, and then after a year or two, it kind of fizzles out. >> Yes. >> Give us a little bit behind the scenes, as to, y'know is it because networking people are worried about their jobs and they're getting on-board? Is it, y'know, I know part of it is your team and the ecosystem you've built here. But, give is some of the reasons why this has succeeded when so many other have, kind of, come and gone. >> Yeah well, I mean we're very fortunate that we've kind of executed in a way that it has continued to be here and we know that's really hard to do. It takes executive support, it takes the troops, it takes fighting anti-bodies, and kind of all of that kind of stuff. But I think, like, the key has been that we've been working with the community. When we had that first DevNet Zone, that first developer conference at Cisco Live four years ago, people came. And that told Cisco something, right? And then as we've continued to build it out, we've actually been not doing it as a silo within Cisco. We've been doing it with our sales organization, with our partner organization, we've been doing it with our ecosystem and our partners and out there. We've just continuously been doing it based on what their needs are. >> And Suzie, I love that, because there are some of the events I saw, they were like, "Well, the developer "is this special unicorn", and we're gonna have this special area, it's velvet rope, we're gonna treat 'em really well. But, this is the first thing you see when you come in, you're very approachable. The line I've heard from your team is, "We are going to meet them where they are." There are no, y'know, "Gosh I haven't "touched programming in 20 years." No, no, no, you're fine, you're good come on in. I'm not sure if I'm really (mumbles). Well you're not programming, you're coding. So, I think that's part of the success, is these people. Y'know, this is their careers, and you're giving them that path forward. >> It is, and when we look at like, developer programs, you'd think it would be easy to start a developer program. But, there's no formula for it, y'know? And when we did it for Cisco, like as we've grown this, it depends on the products that we have, it depends on the community that we have, the types of solutions, what our customers want. And basically what happens is, we did have a core set of networkers who are scared. And we, instead of making DevNet the elite place for the elite developers, we said it is the place to bring in the community. We're gonna be welcoming, we're bringing them in on the journey, because they're the ones who need to be there. And so we've really tried this more open approach. And if you look at Cisco's community of networkers, they're amazing, like, they are developing and installing and operating networks around the World in every country. They've been dedicated, but they are scared of that transition to software and programmability. And they've been dedicated to us, we're dedicated to them, getting to that next level. >> You just did a good job of bringing that tribe kind of mentality and co-development, co-creation, people who are learning. So you have first time learners kicking the tires on coding and growing and experts. So Cisco Champions coming in; Powerhouse developers. >> Yeah >> Not Cisco employees, it's Cisco Champions, and so a nice balance. So that's a good sign of success. >> And you're right, that's key because it's not just, like just beginners. I mean, first of all, there is a very large stage of new people who are just coming in and then wanting to get started and that's awesome. And in addition, very advanced folks, who are like, y'know, just the most advanced developer you'd find, who also has networking expertise. And then of course, the app developers. We're talking to app developers and cloud developers and DevOps pros, and they're coming in as well. >> Yea, and Suzie you bring up a great point. Cause one of the challenges when you have the cool new innovation stuff, is the business, like well how does that connect back? So help connect the dots, we heard Chuck Robbins on stage. Not only was it just DevNet and 500,000 but the new products that are coming out just tie right into it. >> It's crazy, like yea, it's awesome. Because what happens is, programmability, Cisco, is building programmability into our entire portfolio. It's not that we have one product that has API's, I mean that's where we were a few years ago. But now we look... Our enterprise networking products, y'know, for the data center, for service provider, for wireless. All of those products are programmable. Our security products are programmable. IoT, collaboration, our entire portfolio is now programmable, so it gives you this kind of whole portfolio of programmability to play with, and that cross-domain. Who covers that many domains? And that's really powerful. When we take a look at the programmability, it was like for the network devices themselves. Like those have Asics that are programmable. So if there's like a new protocol that comes up to handle IoT things, we can actually re-program the Asics to get that going at line rates. You can do like, on-board application hosting on those network devices. We have controller levels, so you can hit the network, and then now you have like analytics and insights that you can do to pull out information from the network, and then be able to, y'know, operate at that level as well. >> So a strategic advantage architecturally for Cisco, certainly in the network side and scaling up at the stack with Kubernetes and (mumbles). We saw Google on-stage, kinda giving an indicator of where it's going. I want to ask you about the culture question for DevNet. Obviously people are fascinated with the success of DevNet, we've been great to follow the success through your journey and being part of it. But for the folks that are now seeing the success, and want to join: What can they expect, if I join the DevNet mission? What's the expectation? What's gonna be the vibe? What would you share to someone watching, that's gonna jump in and join the journey, what can they expect? >> Well, I think that first of all, it's going to be very welcoming. Like, they're gonna feel welcome. And I'm just proud of my team, because people come in and they actually say, "Wow, sometimes you go to developer conferences "and it's a little bit intimidating." And yea, you might be intimidated, but here you're going to feel welcome. Because, y'know, we really want things to happen. And then there's gonna be this kind of like, intrigue in terms of what you can build. Because what we're building is different. It's not a well known area, like everyone knows how to build apps for a mobile device. People don't know how to build applications for programmable infrastructure. Like, the fact that hey, your wireless access points now give you location and proximity information. I can write an indoor location app. Sounds simple, but it's awesome. >> Connect a camera to it. >> It's amazing, right? >> Hello! >> And then what happens is, as you're doing that, you have like, connect a camera, you're like put a Playstation into a hospital... The Children's Hospital of L.A came and spoke, and they were talking about the business problem. They had a patient, who was very sick, a young boy. And his wish was to have Playstation so he could play it. And then they had to go to their networkers cause you don't put Playstations in hospitals. They had to make that happen and intent-based networking lets you make that wish, and then activate that in the network, that's now a programmable infrastructure. So the types of problems that you can solve are different, it's amazing. >> The new apps are coming out and you're creating a new, first generation green field of networked apps. >> Yes. (chuckles heartily) >> Like what iPhone did for mobile apps, you guys are doing for networks. >> That's right, that's right. >> So that's awesome, it's super cool. Programmable infrastructure, all DevOps kinda geeky stuff. For the next steps, as you guys are now at the beginning of the next inflection point. >> Yes. >> What're you guys focused on? What's happening with the team? What's happening with some of the initiatives you're doing? Also demos get better and better. The training classes are still going on. What's your focus? >> So with some of the things that are happening now, which is... So we've hit this milestone of half a million developers. But what does that mean? What that means is that, we have half a million people who can use network API's. What that means also, is that they're contributing code. So it's no longer just, "Here I'm gonna help "you use your API", but now it's also like, they're contributors back. And what we're doing, is we're actually embracing that and making that part of the innovation model for networking. So, you're not just taking Cisco's platforms and the innovation there, which is of course growing tremendously, but now you can also add in innovation by the community. And I know it's a straight forward concept for software. It's not a straightforward concept for networking and infrastructure. >> To bring an open-source ethos, to code sharing, co-contributing. >> Exactly, and something that we've released is code exchange, definite code exchange. And what it is, is just a list of curated software. Software that's out of GitHub, that works for our platforms, y'know. But the thing that developers are always like, "Okay there's a lot of software out there, "which one should I use?" and then basically giving them like, the curated list of here's the stuff that you can use. >> So Suzie, it's been fun to watch the transformation of Cisco overall. As we look at... Before, we used to measure in boxes and ports. What's the measurement internally? When you talk about saying, "Okay how are we doing "on our journey to become a software company?" Give us a little insight as to internally how Cisco measures that. >> The way that we measure that now is, we're talking to our customers and our partners and their adoption of API's, of programmability, their ability to execute on that and to be successful in this business. And so, it's really an external looking view. So it's all just like okay, how much do they get it? How much can they use it? How much are they building the skills? So it's really looking at the success of the community and being able to build the skills and use these products and build solutions with them. >> Suzie, congratulations on continuing growing, hitting a major milestone, 500,000 developers, half a million developers, that's a real community. It's just the beginning now, it's the start line. >> (chuckling) The start line, it is. >> One finish line is another start line. >> It is a start line, it's absolutely the start line. >> And you guys had a great event last night at the Mango party, the Mango Cafe. Talk about that, you had a celebration. Turns out a lot of people showed up. It was supposed to be a little private party. >> It was a little private party, yea. So we, y'know, just wanted to thank the team and thank our community. Because, quite honestly, to get to this half a million it wasn't just the people who work for me who got it there. It's the fact that, there's of course our team who's very dedicated to that, but then it's our partners. It's even you guys, right? It's our partners who have like... I understand this mission, I'm gonna jump in, I'm gonna help it happen. It's our systems engineers, it's our partners, it's our innovation folks, it's people from the community who understand the mission and have joined in to push it forward. So we had this party last night at Mango Cafe, you guys were there. The people were callin it kinda the best one. It's really just appreciation for our community and what they've done to get it there. Because it's not us, it's our community who've done it. >> This is the open ethos. Cisco becoming open. What's it like to be on the inside and seeing Cisco open up like this? >> It's, I mean, it's amazing. And what's amazing is like, when I started DevNet you'd think like okay, "I'm gonna run a developer program." The thing that surprises me is just, how hurtful it is to so many people. Like, people, they find a path. They see a new opportunity, they figure out a new way they wanna advance their businesses and their careers. And it's like, all heart. And that's how it grew. Like with the resources, it's just because people who had felt this heart and this connection into this mission and drive, they're taking it to the next level so it's amazing >> Like open-source software, people love to be part of a great project. >> It is, it is. >> And DevNet certainly is. And DevNet Create. Don't forget DevNet Create is your other event that bring the cloud native world with the networking world together. >> It is. >> Great project. >> You were with us at DevNet Create and that's where it's this mixing of communities of like, the app developers with the networkers who are getting out there. And what's funny is, we didn't know how those communities would interact. And they're mixing, they're getting it. They're just like "Okay, I have this location software, "I need to work together with the guys "who are gonna install the network and then "we can make this amazing experience." And they're mixing and when they do it the right things happening. >> Very complimentary, there's love going wild. >> App guys love the network guys to take care of the network and the network guys love the app guys that take care of the apps. >> Exactly! Exactly. >> It's a win-win. Great stuff, congratulations. Again, a new way to program. Just like we saw the iPhone creating the app store. Networking now is programmable. We expect to see a lot of great creativity, new problems, new things being created. And that's an opportunity for all. We're here at theCUBE bringing you all the action from the DevNet Zone at Cisco Live. More live coverage. Day three, stay with us, I'm John Furrier with Stu Miniman, we'll be right back. (upbeat music)

>> There's a movement that's sweeping across businesses everywhere here in this country and around the world. And it's all about data. Today businesses are being inundated with data. To the tune of over two and a half million gigabytes that'll be generated in the next 60 seconds alone. What do you do with all that data? To extract insights you typically turn to a data scientist. But not necessarily anymore. At least not exclusively. Today the ability to extract value from data is becoming a shared mission. A team effort that spans the organization extending far more widely than ever before. Today, data science is being democratized. >> Data Sciences for All: It's a Whole New Game. >> Welcome everyone, I'm Katie Linendoll. I'm a technology expert writer and I love reporting on all things tech. My fascination with tech started very young. I began coding when I was 12. Received my networking certs by 18 and a degree in IT and new media from Rochester Institute of Technology. So as you can tell, technology has always been a sure passion of mine. Having grown up in the digital age, I love having a career that keeps me at the forefront of science and technology innovations. I spend equal time in the field being hands on as I do on my laptop conducting in depth research. Whether I'm diving underwater with NASA astronauts, witnessing the new ways which mobile technology can help rebuild the Philippine's economy in the wake of super typhoons, or sharing a first look at the newest iPhones on The Today Show, yesterday, I'm always on the hunt for the latest and greatest tech stories. And that's what brought me here. I'll be your host for the next hour and as we explore the new phenomenon that is taking businesses around the world by storm. And data science continues to become democratized and extends beyond the domain of the data scientist. And why there's also a mandate for all of us to become data literate. Now that data science for all drives our AI culture. And we're going to be able to take to the streets and go behind the scenes as we uncover the factors that are fueling this phenomenon and giving rise to a movement that is reshaping how businesses leverage data. And putting organizations on the road to AI. So coming up, I'll be doing interviews with data scientists. We'll see real world demos and take a look at how IBM is changing the game with an open data science platform. We'll also be joined by legendary statistician Nate Silver, founder and editor-in-chief of FiveThirtyEight. Who will shed light on how a data driven mindset is changing everything from business to our culture. We also have a few people who are joining us in our studio, so thank you guys for joining us. Come on, I can do better than that, right? Live studio audience, the fun stuff. And for all of you during the program, I want to remind you to join that conversation on social media using the hashtag DSforAll, it's data science for all. Share your thoughts on what data science and AI means to you and your business. And, let's dive into a whole new game of data science. Now I'd like to welcome my co-host General Manager IBM Analytics, Rob Thomas. >> Hello, Katie. >> Come on guys. >> Yeah, seriously. >> No one's allowed to be quiet during this show, okay? >> Right. >> Or, I'll start calling people out. So Rob, thank you so much. I think you know this conversation, we're calling it a data explosion happening right now. And it's nothing new. And when you and I chatted about it. You've been talking about this for years. You have to ask, is this old news at this point? >> Yeah, I mean, well first of all, the data explosion is not coming, it's here. And everybody's in the middle of it right now. What is different is the economics have changed. And the scale and complexity of the data that organizations are having to deal with has changed. And to this day, 80% of the data in the world still sits behind corporate firewalls. So, that's becoming a problem. It's becoming unmanageable. IT struggles to manage it. The business can't get everything they need. Consumers can't consume it when they want. So we have a challenge here. >> It's challenging in the world of unmanageable. Crazy complexity. If I'm sitting here as an IT manager of my business, I'm probably thinking to myself, this is incredibly frustrating. How in the world am I going to get control of all this data? And probably not just me thinking it. Many individuals here as well. >> Yeah, indeed. Everybody's thinking about how am I going to put data to work in my organization in a way I haven't done before. Look, you've got to have the right expertise, the right tools. The other thing that's happening in the market right now is clients are dealing with multi cloud environments. So data behind the firewall in private cloud, multiple public clouds. And they have to find a way. How am I going to pull meaning out of this data? And that brings us to data science and AI. That's how you get there. >> I understand the data science part but I think we're all starting to hear more about AI. And it's incredible that this buzz word is happening. How do businesses adopt to this AI growth and boom and trend that's happening in this world right now? >> Well, let me define it this way. Data science is a discipline. And machine learning is one technique. And then AI puts both machine learning into practice and applies it to the business. So this is really about how getting your business where it needs to go. And to get to an AI future, you have to lay a data foundation today. I love the phrase, "there's no AI without IA." That means you're not going to get to AI unless you have the right information architecture to start with. >> Can you elaborate though in terms of how businesses can really adopt AI and get started. >> Look, I think there's four things you have to do if you're serious about AI. One is you need a strategy for data acquisition. Two is you need a modern data architecture. Three is you need pervasive automation. And four is you got to expand job roles in the organization. >> Data acquisition. First pillar in this you just discussed. Can we start there and explain why it's so critical in this process? >> Yeah, so let's think about how data acquisition has evolved through the years. 15 years ago, data acquisition was about how do I get data in and out of my ERP system? And that was pretty much solved. Then the mobile revolution happens. And suddenly you've got structured and non-structured data. More than you've ever dealt with. And now you get to where we are today. You're talking terabytes, petabytes of data. >> [Katie] Yottabytes, I heard that word the other day. >> I heard that too. >> Didn't even know what it meant. >> You know how many zeros that is? >> I thought we were in Star Wars. >> Yeah, I think it's a lot of zeroes. >> Yodabytes, it's new. >> So, it's becoming more and more complex in terms of how you acquire data. So that's the new data landscape that every client is dealing with. And if you don't have a strategy for how you acquire that and manage it, you're not going to get to that AI future. >> So a natural segue, if you are one of these businesses, how do you build for the data landscape? >> Yeah, so the question I always hear from customers is we need to evolve our data architecture to be ready for AI. And the way I think about that is it's really about moving from static data repositories to more of a fluid data layer. >> And we continue with the architecture. New data architecture is an interesting buzz word to hear. But it's also one of the four pillars. So if you could dive in there. >> Yeah, I mean it's a new twist on what I would call some core data science concepts. For example, you have to leverage tools with a modern, centralized data warehouse. But your data warehouse can't be stagnant to just what's right there. So you need a way to federate data across different environments. You need to be able to bring your analytics to the data because it's most efficient that way. And ultimately, it's about building an optimized data platform that is designed for data science and AI. Which means it has to be a lot more flexible than what clients have had in the past. >> All right. So we've laid out what you need for driving automation. But where does the machine learning kick in? >> Machine learning is what gives you the ability to automate tasks. And I think about machine learning. It's about predicting and automating. And this will really change the roles of data professionals and IT professionals. For example, a data scientist cannot possibly know every algorithm or every model that they could use. So we can automate the process of algorithm selection. Another example is things like automated data matching. Or metadata creation. Some of these things may not be exciting but they're hugely practical. And so when you think about the real use cases that are driving return on investment today, it's things like that. It's automating the mundane tasks. >> Let's go ahead and come back to something that you mentioned earlier because it's fascinating to be talking about this AI journey, but also significant is the new job roles. And what are those other participants in the analytics pipeline? >> Yeah I think we're just at the start of this idea of new job roles. We have data scientists. We have data engineers. Now you see machine learning engineers. Application developers. What's really happening is that data scientists are no longer allowed to work in their own silo. And so the new job roles is about how does everybody have data first in their mind? And then they're using tools to automate data science, to automate building machine learning into applications. So roles are going to change dramatically in organizations. >> I think that's confusing though because we have several organizations who saying is that highly specialized roles, just for data science? Or is it applicable to everybody across the board? >> Yeah, and that's the big question, right? Cause everybody's thinking how will this apply? Do I want this to be just a small set of people in the organization that will do this? But, our view is data science has to for everybody. It's about bring data science to everybody as a shared mission across the organization. Everybody in the company has to be data literate. And participate in this journey. >> So overall, group effort, has to be a common goal, and we all need to be data literate across the board. >> Absolutely. >> Done deal. But at the end of the day, it's kind of not an easy task. >> It's not. It's not easy but it's maybe not as big of a shift as you would think. Because you have to put data in the hands of people that can do something with it. So, it's very basic. Give access to data. Data's often locked up in a lot of organizations today. Give people the right tools. Embrace the idea of choice or diversity in terms of those tools. That gets you started on this path. >> It's interesting to hear you say essentially you need to train everyone though across the board when it comes to data literacy. And I think people that are coming into the work force don't necessarily have a background or a degree in data science. So how do you manage? >> Yeah, so in many cases that's true. I will tell you some universities are doing amazing work here. One example, University of California Berkeley. They offer a course for all majors. So no matter what you're majoring in, you have a course on foundations of data science. How do you bring data science to every role? So it's starting to happen. We at IBM provide data science courses through CognitiveClass.ai. It's for everybody. It's free. And look, if you want to get your hands on code and just dive right in, you go to datascience.ibm.com. The key point is this though. It's more about attitude than it is aptitude. I think anybody can figure this out. But it's about the attitude to say we're putting data first and we're going to figure out how to make this real in our organization. >> I also have to give a shout out to my alma mater because I have heard that there is an offering in MS in data analytics. And they are always on the forefront of new technologies and new majors and on trend. And I've heard that the placement behind those jobs, people graduating with the MS is high. >> I'm sure it's very high. >> So go Tigers. All right, tangential. Let me get back to something else you touched on earlier because you mentioned that a number of customers ask you how in the world do I get started with AI? It's an overwhelming question. Where do you even begin? What do you tell them? >> Yeah, well things are moving really fast. But the good thing is most organizations I see, they're already on the path, even if they don't know it. They might have a BI practice in place. They've got data warehouses. They've got data lakes. Let me give you an example. AMC Networks. They produce a lot of the shows that I'm sure you watch Katie. >> [Katie] Yes, Breaking Bad, Walking Dead, any fans? >> [Rob] Yeah, we've got a few. >> [Katie] Well you taught me something I didn't even know. Because it's amazing how we have all these different industries, but yet media in itself is impacted too. And this is a good example. >> Absolutely. So, AMC Networks, think about it. They've got ads to place. They want to track viewer behavior. What do people like? What do they dislike? So they have to optimize every aspect of their business from marketing campaigns to promotions to scheduling to ads. And their goal was transform data into business insights and really take the burden off of their IT team that was heavily burdened by obviously a huge increase in data. So their VP of BI took the approach of using machine learning to process large volumes of data. They used a platform that was designed for AI and data processing. It's the IBM analytics system where it's a data warehouse, data science tools are built in. It has in memory data processing. And just like that, they were ready for AI. And they're already seeing that impact in their business. >> Do you think a movement of that nature kind of presses other media conglomerates and organizations to say we need to be doing this too? >> I think it's inevitable that everybody, you're either going to be playing, you're either going to be leading, or you'll be playing catch up. And so, as we talk to clients we think about how do you start down this path now, even if you have to iterate over time? Because otherwise you're going to wake up and you're going to be behind. >> One thing worth noting is we've talked about analytics to the data. It's analytics first to the data, not the other way around. >> Right. So, look. We as a practice, we say you want to bring data to where the data sits. Because it's a lot more efficient that way. It gets you better outcomes in terms of how you train models and it's more efficient. And we think that leads to better outcomes. Other organization will say, "Hey move the data around." And everything becomes a big data movement exercise. But once an organization has started down this path, they're starting to get predictions, they want to do it where it's really easy. And that means analytics applied right where the data sits. >> And worth talking about the role of the data scientist in all of this. It's been called the hot job of the decade. And a Harvard Business Review even dubbed it the sexiest job of the 21st century. >> Yes. >> I want to see this on the cover of Vogue. Like I want to see the first data scientist. Female preferred, on the cover of Vogue. That would be amazing. >> Perhaps you can. >> People agree. So what changes for them? Is this challenging in terms of we talk data science for all. Where do all the data science, is it data science for everyone? And how does it change everything? >> Well, I think of it this way. AI gives software super powers. It really does. It changes the nature of software. And at the center of that is data scientists. So, a data scientist has a set of powers that they've never had before in any organization. And that's why it's a hot profession. Now, on one hand, this has been around for a while. We've had actuaries. We've had statisticians that have really transformed industries. But there are a few things that are new now. We have new tools. New languages. Broader recognition of this need. And while it's important to recognize this critical skill set, you can't just limit it to a few people. This is about scaling it across the organization. And truly making it accessible to all. >> So then do we need more data scientists? Or is this something you train like you said, across the board? >> Well, I think you want to do a little bit of both. We want more. But, we can also train more and make the ones we have more productive. The way I think about it is there's kind of two markets here. And we call it clickers and coders. >> [Katie] I like that. That's good. >> So, let's talk about what that means. So clickers are basically somebody that wants to use tools. Create models visually. It's drag and drop. Something that's very intuitive. Those are the clickers. Nothing wrong with that. It's been valuable for years. There's a new crop of data scientists. They want to code. They want to build with the latest open source tools. They want to write in Python or R. These are the coders. And both approaches are viable. Both approaches are critical. Organizations have to have a way to meet the needs of both of those types. And there's not a lot of things available today that do that. >> Well let's keep going on that. Because I hear you talking about the data scientists role and how it's critical to success, but with the new tools, data science and analytics skills can extend beyond the domain of just the data scientist. >> That's right. So look, we're unifying coders and clickers into a single platform, which we call IBM Data Science Experience. And as the demand for data science expertise grows, so does the need for these kind of tools. To bring them into the same environment. And my view is if you have the right platform, it enables the organization to collaborate. And suddenly you've changed the nature of data science from an individual sport to a team sport. >> So as somebody that, my background is in IT, the question is really is this an additional piece of what IT needs to do in 2017 and beyond? Or is it just another line item to the budget? >> So I'm afraid that some people might view it that way. As just another line item. But, I would challenge that and say data science is going to reinvent IT. It's going to change the nature of IT. And every organization needs to think about what are the skills that are critical? How do we engage a broader team to do this? Because once they get there, this is the chance to reinvent how they're performing IT. >> [Katie] Challenging or not? >> Look it's all a big challenge. Think about everything IT organizations have been through. Some of them were late to things like mobile, but then they caught up. Some were late to cloud, but then they caught up. I would just urge people, don't be late to data science. Use this as your chance to reinvent IT. Start with this notion of clickers and coders. This is a seminal moment. Much like mobile and cloud was. So don't be late. >> And I think it's critical because it could be so costly to wait. And Rob and I were even chatting earlier how data analytics is just moving into all different kinds of industries. And I can tell you even personally being effected by how important the analysis is in working in pediatric cancer for the last seven years. I personally implement virtual reality headsets to pediatric cancer hospitals across the country. And it's great. And it's working phenomenally. And the kids are amazed. And the staff is amazed. But the phase two of this project is putting in little metrics in the hardware that gather the breathing, the heart rate to show that we have data. Proof that we can hand over to the hospitals to continue making this program a success. So just in-- >> That's a great example. >> An interesting example. >> Saving lives? >> Yes. >> That's also applying a lot of what we talked about. >> Exciting stuff in the world of data science. >> Yes. Look, I just add this is an existential moment for every organization. Because what you do in this area is probably going to define how competitive you are going forward. And think about if you don't do something. What if one of your competitors goes and creates an application that's more engaging with clients? So my recommendation is start small. Experiment. Learn. Iterate on projects. Define the business outcomes. Then scale up. It's very doable. But you've got to take the first step. >> First step always critical. And now we're going to get to the fun hands on part of our story. Because in just a moment we're going to take a closer look at what data science can deliver. And where organizations are trying to get to. All right. Thank you Rob and now we've been joined by Siva Anne who is going to help us navigate this demo. First, welcome Siva. Give him a big round of applause. Yeah. All right, Rob break down what we're going to be looking at. You take over this demo. >> All right. So this is going to be pretty interesting. So Siva is going to take us through. So he's going to play the role of a financial adviser. Who wants to help better serve clients through recommendations. And I'm going to really illustrate three things. One is how do you federate data from multiple data sources? Inside the firewall, outside the firewall. How do you apply machine learning to predict and to automate? And then how do you move analytics closer to your data? So, what you're seeing here is a custom application for an investment firm. So, Siva, our financial adviser, welcome. So you can see at the top, we've got market data. We pulled that from an external source. And then we've got Siva's calendar in the middle. He's got clients on the right side. So page down, what else do you see down there Siva? >> [Siva] I can see the recent market news. And in here I can see that JP Morgan is calling for a US dollar rebound in the second half of the year. And, I have upcoming meeting with Leo Rakes. I can get-- >> [Rob] So let's go in there. Why don't you click on Leo Rakes. So, you're sitting at your desk, you're deciding how you're going to spend the day. You know you have a meeting with Leo. So you click on it. You immediately see, all right, so what do we know about him? We've got data governance implemented. So we know his age, we know his degree. We can see he's not that aggressive of a trader. Only six trades in the last few years. But then where it gets interesting is you go to the bottom. You start to see predicted industry affinity. Where did that come from? How do we have that? >> [Siva] So these green lines and red arrows here indicate the trending affinity of Leo Rakes for particular industry stocks. What we've done here is we've built machine learning models using customer's demographic data, his stock portfolios, and browsing behavior to build a model which can predict his affinity for a particular industry. >> [Rob] Interesting. So, I like to think of this, we call it celebrity experiences. So how do you treat every customer like they're a celebrity? So to some extent, we're reading his mind. Because without asking him, we know that he's going to have an affinity for auto stocks. So we go down. Now we look at his portfolio. You can see okay, he's got some different holdings. He's got Amazon, Google, Apple, and then he's got RACE, which is the ticker for Ferrari. You can see that's done incredibly well. And so, as a financial adviser, you look at this and you say, all right, we know he loves auto stocks. Ferrari's done very well. Let's create a hedge. Like what kind of security would interest him as a hedge against his position for Ferrari? Could we go figure that out? >> [Siva] Yes. Given I know that he's gotten an affinity for auto stocks, and I also see that Ferrari has got some terminus gains, I want to lock in these gains by hedging. And I want to do that by picking a auto stock which has got negative correlation with Ferrari. >> [Rob] So this is where we get to the idea of in database analytics. Cause you start clicking that and immediately we're getting instant answers of what's happening. So what did we find here? We're going to compare Ferrari and Honda. >> [Siva] I'm going to compare Ferrari with Honda. And what I see here instantly is that Honda has got a negative correlation with Ferrari, which makes it a perfect mix for his stock portfolio. Given he has an affinity for auto stocks and it correlates negatively with Ferrari. >> [Rob] These are very powerful tools at the hand of a financial adviser. You think about it. As a financial adviser, you wouldn't think about federating data, machine learning, pretty powerful. >> [Siva] Yes. So what we have seen here is that using the common SQL engine, we've been able to federate queries across multiple data sources. Db2 Warehouse in the cloud, IBM's Integrated Analytic System, and Hortonworks powered Hadoop platform for the new speeds. We've been able to use machine learning to derive innovative insights about his stock affinities. And drive the machine learning into the appliance. Closer to where the data resides to deliver high performance analytics. >> [Rob] At scale? >> [Siva] We're able to run millions of these correlations across stocks, currency, other factors. And even score hundreds of customers for their affinities on a daily basis. >> That's great. Siva, thank you for playing the role of financial adviser. So I just want to recap briefly. Cause this really powerful technology that's really simple. So we federated, we aggregated multiple data sources from all over the web and internal systems. And public cloud systems. Machine learning models were built that predicted Leo's affinity for a certain industry. In this case, automotive. And then you see when you deploy analytics next to your data, even a financial adviser, just with the click of a button is getting instant answers so they can go be more productive in their next meeting. This whole idea of celebrity experiences for your customer, that's available for everybody, if you take advantage of these types of capabilities. Katie, I'll hand it back to you. >> Good stuff. Thank you Rob. Thank you Siva. Powerful demonstration on what we've been talking about all afternoon. And thank you again to Siva for helping us navigate. Should be give him one more round of applause? We're going to be back in just a moment to look at how we operationalize all of this data. But in first, here's a message from me. If you're a part of a line of business, your main fear is disruption. You know data is the new goal that can create huge amounts of value. So does your competition. And they may be beating you to it. You're convinced there are new business models and revenue sources hidden in all the data. You just need to figure out how to leverage it. But with the scarcity of data scientists, you really can't rely solely on them. You may need more people throughout the organization that have the ability to extract value from data. And as a data science leader or data scientist, you have a lot of the same concerns. You spend way too much time looking for, prepping, and interpreting data and waiting for models to train. You know you need to operationalize the work you do to provide business value faster. What you want is an easier way to do data prep. And rapidly build models that can be easily deployed, monitored and automatically updated. So whether you're a data scientist, data science leader, or in a line of business, what's the solution? What'll it take to transform the way you work? That's what we're going to explore next. All right, now it's time to delve deeper into the nuts and bolts. The nitty gritty of operationalizing data science and creating a data driven culture. How do you actually do that? Well that's what these experts are here to share with us. I'm joined by Nir Kaldero, who's head of data science at Galvanize, which is an education and training organization. Tricia Wang, who is co-founder of Sudden Compass, a consultancy that helps companies understand people with data. And last, but certainly not least, Michael Li, founder and CEO of Data Incubator, which is a data science train company. All right guys. Shall we get right to it? >> All right. >> So data explosion happening right now. And we are seeing it across the board. I just shared an example of how it's impacting my philanthropic work in pediatric cancer. But you guys each have so many unique roles in your business life. How are you seeing it just blow up in your fields? Nir, your thing? >> Yeah, for example like in Galvanize we train many Fortune 500 companies. And just by looking at the demand of companies that wants us to help them go through this digital transformation is mind-blowing. Data point by itself. >> Okay. Well what we're seeing what's going on is that data science like as a theme, is that it's actually for everyone now. But what's happening is that it's actually meeting non technical people. But what we're seeing is that when non technical people are implementing these tools or coming at these tools without a base line of data literacy, they're often times using it in ways that distance themselves from the customer. Because they're implementing data science tools without a clear purpose, without a clear problem. And so what we do at Sudden Compass is that we work with companies to help them embrace and understand the complexity of their customers. Because often times they are misusing data science to try and flatten their understanding of the customer. As if you can just do more traditional marketing. Where you're putting people into boxes. And I think the whole ROI of data is that you can now understand people's relationships at a much more complex level at a greater scale before. But we have to do this with basic data literacy. And this has to involve technical and non technical people. >> Well you can have all the data in the world, and I think it speaks to, if you're not doing the proper movement with it, forget it. It means nothing at the same time. >> No absolutely. I mean, I think that when you look at the huge explosion in data, that comes with it a huge explosion in data experts. Right, we call them data scientists, data analysts. And sometimes they're people who are very, very talented, like the people here. But sometimes you have people who are maybe re-branding themselves, right? Trying to move up their title one notch to try to attract that higher salary. And I think that that's one of the things that customers are coming to us for, right? They're saying, hey look, there are a lot of people that call themselves data scientists, but we can't really distinguish. So, we have sort of run a fellowship where you help companies hire from a really talented group of folks, who are also truly data scientists and who know all those kind of really important data science tools. And we also help companies internally. Fortune 500 companies who are looking to grow that data science practice that they have. And we help clients like McKinsey, BCG, Bain, train up their customers, also their clients, also their workers to be more data talented. And to build up that data science capabilities. >> And Nir, this is something you work with a lot. A lot of Fortune 500 companies. And when we were speaking earlier, you were saying many of these companies can be in a panic. >> Yeah. >> Explain that. >> Yeah, so you know, not all Fortune 500 companies are fully data driven. And we know that the winners in this fourth industrial revolution, which I like to call the machine intelligence revolution, will be companies who navigate and transform their organization to unlock the power of data science and machine learning. And the companies that are not like that. Or not utilize data science and predictive power well, will pretty much get shredded. So they are in a panic. >> Tricia, companies have to deal with data behind the firewall and in the new multi cloud world. How do organizations start to become driven right to the core? >> I think the most urgent question to become data driven that companies should be asking is how do I bring the complex reality that our customers are experiencing on the ground in to a corporate office? Into the data models. So that question is critical because that's how you actually prevent any big data disasters. And that's how you leverage big data. Because when your data models are really far from your human models, that's when you're going to do things that are really far off from how, it's going to not feel right. That's when Tesco had their terrible big data disaster that they're still recovering from. And so that's why I think it's really important to understand that when you implement big data, you have to further embrace thick data. The qualitative, the emotional stuff, that is difficult to quantify. But then comes the difficult art and science that I think is the next level of data science. Which is that getting non technical and technical people together to ask how do we find those unknown nuggets of insights that are difficult to quantify? Then, how do we do the next step of figuring out how do you mathematically scale those insights into a data model? So that actually is reflective of human understanding? And then we can start making decisions at scale. But you have to have that first. >> That's absolutely right. And I think that when we think about what it means to be a data scientist, right? I always think about it in these sort of three pillars. You have the math side. You have to have that kind of stats, hardcore machine learning background. You have the programming side. You don't work with small amounts of data. You work with large amounts of data. You've got to be able to type the code to make those computers run. But then the last part is that human element. You have to understand the domain expertise. You have to understand what it is that I'm actually analyzing. What's the business proposition? And how are the clients, how are the users actually interacting with the system? That human element that you were talking about. And I think having somebody who understands all of those and not just in isolation, but is able to marry that understanding across those different topics, that's what makes a data scientist. >> But I find that we don't have people with those skill sets. And right now the way I see teams being set up inside companies is that they're creating these isolated data unicorns. These data scientists that have graduated from your programs, which are great. But, they don't involve the people who are the domain experts. They don't involve the designers, the consumer insight people, the people, the salespeople. The people who spend time with the customers day in and day out. Somehow they're left out of the room. They're consulted, but they're not a stakeholder. >> Can I actually >> Yeah, yeah please. >> Can I actually give a quick example? So for example, we at Galvanize train the executives and the managers. And then the technical people, the data scientists and the analysts. But in order to actually see all of the RY behind the data, you also have to have a creative fluid conversation between non technical and technical people. And this is a major trend now. And there's a major gap. And we need to increase awareness and kind of like create a new, kind of like environment where technical people also talks seamlessly with non technical ones. >> [Tricia] We call-- >> That's one of the things that we see a lot. Is one of the trends in-- >> A major trend. >> data science training is it's not just for the data science technical experts. It's not just for one type of person. So a lot of the training we do is sort of data engineers. People who are more on the software engineering side learning more about the stats of math. And then people who are sort of traditionally on the stat side learning more about the engineering. And then managers and people who are data analysts learning about both. >> Michael, I think you said something that was of interest too because I think we can look at IBM Watson as an example. And working in healthcare. The human component. Because often times we talk about machine learning and AI, and data and you get worried that you still need that human component. Especially in the world of healthcare. And I think that's a very strong point when it comes to the data analysis side. Is there any particular example you can speak to of that? >> So I think that there was this really excellent paper a while ago talking about all the neuro net stuff and trained on textual data. So looking at sort of different corpuses. And they found that these models were highly, highly sexist. They would read these corpuses and it's not because neuro nets themselves are sexist. It's because they're reading the things that we write. And it turns out that we write kind of sexist things. And they would sort of find all these patterns in there that were sort of latent, that had a lot of sort of things that maybe we would cringe at if we sort of saw. And I think that's one of the really important aspects of the human element, right? It's being able to come in and sort of say like, okay, I know what the biases of the system are, I know what the biases of the tools are. I need to figure out how to use that to make the tools, make the world a better place. And like another area where this comes up all the time is lending, right? So the federal government has said, and we have a lot of clients in the financial services space, so they're constantly under these kind of rules that they can't make discriminatory lending practices based on a whole set of protected categories. Race, sex, gender, things like that. But, it's very easy when you train a model on credit scores to pick that up. And then to have a model that's inadvertently sexist or racist. And that's where you need the human element to come back in and say okay, look, you're using the classic example would be zip code, you're using zip code as a variable. But when you look at it, zip codes actually highly correlated with race. And you can't do that. So you may inadvertently by sort of following the math and being a little naive about the problem, inadvertently introduce something really horrible into a model and that's where you need a human element to sort of step in and say, okay hold on. Slow things down. This isn't the right way to go. >> And the people who have -- >> I feel like, I can feel her ready to respond. >> Yes, I'm ready. >> She's like let me have at it. >> And the people here it is. And the people who are really great at providing that human intelligence are social scientists. We are trained to look for bias and to understand bias in data. Whether it's quantitative or qualitative. And I really think that we're going to have less of these kind of problems if we had more integrated teams. If it was a mandate from leadership to say no data science team should be without a social scientist, ethnographer, or qualitative researcher of some kind, to be able to help see these biases. >> The talent piece is actually the most crucial-- >> Yeah. >> one here. If you look about how to enable machine intelligence in organization there are the pillars that I have in my head which is the culture, the talent and the technology infrastructure. And I believe and I saw in working very closely with the Fortune 100 and 200 companies that the talent piece is actually the most important crucial hard to get. >> [Tricia] I totally agree. >> It's absolutely true. Yeah, no I mean I think that's sort of like how we came up with our business model. Companies were basically saying hey, I can't hire data scientists. And so we have a fellowship where we get 2,000 applicants each quarter. We take the top 2% and then we sort of train them up. And we work with hiring companies who then want to hire from that population. And so we're sort of helping them solve that problem. And the other half of it is really around training. Cause with a lot of industries, especially if you're sort of in a more regulated industry, there's a lot of nuances to what you're doing. And the fastest way to develop that data science or AI talent may not necessarily be to hire folks who are coming out of a PhD program. It may be to take folks internally who have a lot of that domain knowledge that you have and get them trained up on those data science techniques. So we've had large insurance companies come to us and say hey look, we hire three or four folks from you a quarter. That doesn't move the needle for us. What we really need is take the thousand actuaries and statisticians that we have and get all of them trained up to become a data scientist and become data literate in this new open source world. >> [Katie] Go ahead. >> All right, ladies first. >> Go ahead. >> Are you sure? >> No please, fight first. >> Go ahead. >> Go ahead Nir. >> So this is actually a trend that we have been seeing in the past year or so that companies kind of like start to look how to upscale and look for talent within the organization. So they can actually move them to become more literate and navigate 'em from analyst to data scientist. And from data scientist to machine learner. So this is actually a trend that is happening already for a year or so. >> Yeah, but I also find that after they've gone through that training in getting people skilled up in data science, the next problem that I get is executives coming to say we've invested in all of this. We're still not moving the needle. We've already invested in the right tools. We've gotten the right skills. We have enough scale of people who have these skills. Why are we not moving the needle? And what I explain to them is look, you're still making decisions in the same way. And you're still not involving enough of the non technical people. Especially from marketing, which is now, the CMO's are much more responsible for driving growth in their companies now. But often times it's so hard to change the old way of marketing, which is still like very segmentation. You know, demographic variable based, and we're trying to move people to say no, you have to understand the complexity of customers and not put them in boxes. >> And I think underlying a lot of this discussion is this question of culture, right? >> Yes. >> Absolutely. >> How do you build a data driven culture? And I think that that culture question, one of the ways that comes up quite often in especially in large, Fortune 500 enterprises, is that they are very, they're not very comfortable with sort of example, open source architecture. Open source tools. And there is some sort of residual bias that that's somehow dangerous. So security vulnerability. And I think that that's part of the cultural challenge that they often have in terms of how do I build a more data driven organization? Well a lot of the talent really wants to use these kind of tools. And I mean, just to give you an example, we are partnering with one of the major cloud providers to sort of help make open source tools more user friendly on their platform. So trying to help them attract the best technologists to use their platform because they want and they understand the value of having that kind of open source technology work seamlessly on their platforms. So I think that just sort of goes to show you how important open source is in this movement. And how much large companies and Fortune 500 companies and a lot of the ones we work with have to embrace that. >> Yeah, and I'm seeing it in our work. Even when we're working with Fortune 500 companies, is that they've already gone through the first phase of data science work. Where I explain it was all about the tools and getting the right tools and architecture in place. And then companies started moving into getting the right skill set in place. Getting the right talent. And what you're talking about with culture is really where I think we're talking about the third phase of data science, which is looking at communication of these technical frameworks so that we can get non technical people really comfortable in the same room with data scientists. That is going to be the phase, that's really where I see the pain point. And that's why at Sudden Compass, we're really dedicated to working with each other to figure out how do we solve this problem now? >> And I think that communication between the technical stakeholders and management and leadership. That's a very critical piece of this. You can't have a successful data science organization without that. >> Absolutely. >> And I think that actually some of the most popular trainings we've had recently are from managers and executives who are looking to say, how do I become more data savvy? How do I figure out what is this data science thing and how do I communicate with my data scientists? >> You guys made this way too easy. I was just going to get some popcorn and watch it play out. >> Nir, last 30 seconds. I want to leave you with an opportunity to, anything you want to add to this conversation? >> I think one thing to conclude is to say that companies that are not data driven is about time to hit refresh and figure how they transition the organization to become data driven. To become agile and nimble so they can actually see what opportunities from this important industrial revolution. Otherwise, unfortunately they will have hard time to survive. >> [Katie] All agreed? >> [Tricia] Absolutely, you're right. >> Michael, Trish, Nir, thank you so much. Fascinating discussion. And thank you guys again for joining us. We will be right back with another great demo. Right after this. >> Thank you Katie. >> Once again, thank you for an excellent discussion. Weren't they great guys? And thank you for everyone who's tuning in on the live webcast. As you can hear, we have an amazing studio audience here. And we're going to keep things moving. I'm now joined by Daniel Hernandez and Siva Anne. And we're going to turn our attention to how you can deliver on what they're talking about using data science experience to do data science faster. >> Thank you Katie. Siva and I are going to spend the next 10 minutes showing you how you can deliver on what they were saying using the IBM Data Science Experience to do data science faster. We'll demonstrate through new features we introduced this week how teams can work together more effectively across the entire analytics life cycle. How you can take advantage of any and all data no matter where it is and what it is. How you could use your favorite tools from open source. And finally how you could build models anywhere and employ them close to where your data is. Remember the financial adviser app Rob showed you? To build an app like that, we needed a team of data scientists, developers, data engineers, and IT staff to collaborate. We do this in the Data Science Experience through a concept we call projects. When I create a new project, I can now use the new Github integration feature. We're doing for data science what we've been doing for developers for years. Distributed teams can work together on analytics projects. And take advantage of Github's version management and change management features. This is a huge deal. Let's explore the project we created for the financial adviser app. As you can see, our data engineer Joane, our developer Rob, and others are collaborating this project. Joane got things started by bringing together the trusted data sources we need to build the app. Taking a closer look at the data, we see that our customer and profile data is stored on our recently announced IBM Integrated Analytics System, which runs safely behind our firewall. We also needed macro economic data, which she was able to find in the Federal Reserve. And she stored it in our Db2 Warehouse on Cloud. And finally, she selected stock news data from NASDAQ.com and landed that in a Hadoop cluster, which happens to be powered by Hortonworks. We added a new feature to the Data Science Experience so that when it's installed with Hortonworks, it automatically uses a need of security and governance controls within the cluster so your data is always secure and safe. Now we want to show you the news data we stored in the Hortonworks cluster. This is the mean administrative console. It's powered by an open source project called Ambari. And here's the news data. It's in parquet files stored in HDFS, which happens to be a distributive file system. To get the data from NASDAQ into our cluster, we used IBM's BigIntegrate and BigQuality to create automatic data pipelines that acquire, cleanse, and ingest that news data. Once the data's available, we use IBM's Big SQL to query that data using SQL statements that are much like the ones we would use for any relation of data, including the data that we have in the Integrated Analytics System and Db2 Warehouse on Cloud. This and the federation capabilities that Big SQL offers dramatically simplifies data acquisition. Now we want to show you how we support a brand new tool that we're excited about. Since we launched last summer, the Data Science Experience has supported Jupyter and R for data analysis and visualization. In this week's update, we deeply integrated another great open source project called Apache Zeppelin. It's known for having great visualization support, advanced collaboration features, and is growing in popularity amongst the data science community. This is an example of Apache Zeppelin and the notebook we created through it to explore some of our data. Notice how wonderful and easy the data visualizations are. Now we want to walk you through the Jupyter notebook we created to explore our customer preference for stocks. We use notebooks to understand and explore data. To identify the features that have some predictive power. Ultimately, we're trying to assess what ultimately is driving customer stock preference. Here we did the analysis to identify the attributes of customers that are likely to purchase auto stocks. We used this understanding to build our machine learning model. For building machine learning models, we've always had tools integrated into the Data Science Experience. But sometimes you need to use tools you already invested in. Like our very own SPSS as well as SAS. Through new import feature, you can easily import those models created with those tools. This helps you avoid vendor lock-in, and simplify the development, training, deployment, and management of all your models. To build the models we used in app, we could have coded, but we prefer a visual experience. We used our customer profile data in the Integrated Analytic System. Used the Auto Data Preparation to cleanse our data. Choose the binary classification algorithms. Let the Data Science Experience evaluate between logistic regression and gradient boosted tree. It's doing the heavy work for us. As you can see here, the Data Science Experience generated performance metrics that show us that the gradient boosted tree is the best performing algorithm for the data we gave it. Once we save this model, it's automatically deployed and available for developers to use. Any application developer can take this endpoint and consume it like they would any other API inside of the apps they built. We've made training and creating machine learning models super simple. But what about the operations? A lot of companies are struggling to ensure their model performance remains high over time. In our financial adviser app, we know that customer data changes constantly, so we need to always monitor model performance and ensure that our models are retrained as is necessary. This is a dashboard that shows the performance of our models and lets our teams monitor and retrain those models so that they're always performing to our standards. So far we've been showing you the Data Science Experience available behind the firewall that we're using to build and train models. Through a new publish feature, you can build models and deploy them anywhere. In another environment, private, public, or anywhere else with just a few clicks. So here we're publishing our model to the Watson machine learning service. It happens to be in the IBM cloud. And also deeply integrated with our Data Science Experience. After publishing and switching to the Watson machine learning service, you can see that our stock affinity and model that we just published is there and ready for use. So this is incredibly important. I just want to say it again. The Data Science Experience allows you to train models behind your own firewall, take advantage of your proprietary and sensitive data, and then deploy those models wherever you want with ease. So summarize what we just showed you. First, IBM's Data Science Experience supports all teams. You saw how our data engineer populated our project with trusted data sets. Our data scientists developed, trained, and tested a machine learning model. Our developers used APIs to integrate machine learning into their apps. And how IT can use our Integrated Model Management dashboard to monitor and manage model performance. Second, we support all data. On premises, in the cloud, structured, unstructured, inside of your firewall, and outside of it. We help you bring analytics and governance to where your data is. Third, we support all tools. The data science tools that you depend on are readily available and deeply integrated. This includes capabilities from great partners like Hortonworks. And powerful tools like our very own IBM SPSS. And fourth, and finally, we support all deployments. You can build your models anywhere, and deploy them right next to where your data is. Whether that's in the public cloud, private cloud, or even on the world's most reliable transaction platform, IBM z. So see for yourself. Go to the Data Science Experience website, take us for a spin. And if you happen to be ready right now, our recently created Data Science Elite Team can help you get started and run experiments alongside you with no charge. Thank you very much. >> Thank you very much Daniel. It seems like a great time to get started. And thanks to Siva for taking us through it. Rob and I will be back in just a moment to add some perspective right after this. All right, once again joined by Rob Thomas. And Rob obviously we got a lot of information here. >> Yes, we've covered a lot of ground. >> This is intense. You got to break it down for me cause I think we zoom out and see the big picture. What better data science can deliver to a business? Why is this so important? I mean we've heard it through and through. >> Yeah, well, I heard it a couple times. But it starts with businesses have to embrace a data driven culture. And it is a change. And we need to make data accessible with the right tools in a collaborative culture because we've got diverse skill sets in every organization. But data driven companies succeed when data science tools are in the hands of everyone. And I think that's a new thought. I think most companies think just get your data scientist some tools, you'll be fine. This is about tools in the hands of everyone. I think the panel did a great job of describing about how we get to data science for all. Building a data culture, making it a part of your everyday operations, and the highlights of what Daniel just showed us, that's some pretty cool features for how organizations can get to this, which is you can see IBM's Data Science Experience, how that supports all teams. You saw data analysts, data scientists, application developer, IT staff, all working together. Second, you saw how we support all tools. And your choice of tools. So the most popular data science libraries integrated into one platform. And we saw some new capabilities that help companies avoid lock-in, where you can import existing models created from specialist tools like SPSS or others. And then deploy them and manage them inside of Data Science Experience. That's pretty interesting. And lastly, you see we continue to build on this best of open tools. Partnering with companies like H2O, Hortonworks, and others. Third, you can see how you use all data no matter where it lives. That's a key challenge every organization's going to face. Private, public, federating all data sources. We announced new integration with the Hortonworks data platform where we deploy machine learning models where your data resides. That's been a key theme. Analytics where the data is. And lastly, supporting all types of deployments. Deploy them in your Hadoop cluster. Deploy them in your Integrated Analytic System. Or deploy them in z, just to name a few. A lot of different options here. But look, don't believe anything I say. Go try it for yourself. Data Science Experience, anybody can use it. Go to datascience.ibm.com and look, if you want to start right now, we just created a team that we call Data Science Elite. These are the best data scientists in the world that will come sit down with you and co-create solutions, models, and prove out a proof of concept. >> Good stuff. Thank you Rob. So you might be asking what does an organization look like that embraces data science for all? And how could it transform your role? I'm going to head back to the office and check it out. Let's start with the perspective of the line of business. What's changed? Well, now you're starting to explore new business models. You've uncovered opportunities for new revenue sources and all that hidden data. And being disrupted is no longer keeping you up at night. As a data science leader, you're beginning to collaborate with a line of business to better understand and translate the objectives into the models that are being built. Your data scientists are also starting to collaborate with the less technical team members and analysts who are working closest to the business problem. And as a data scientist, you stop feeling like you're falling behind. Open source tools are keeping you current. You're also starting to operationalize the work that you do. And you get to do more of what you love. Explore data, build models, put your models into production, and create business impact. All in all, it's not a bad scenario. Thanks. All right. We are back and coming up next, oh this is a special time right now. Cause we got a great guest speaker. New York Magazine called him the spreadsheet psychic and number crunching prodigy who went from correctly forecasting baseball games to correctly forecasting presidential elections. He even invented a proprietary algorithm called PECOTA for predicting future performance by baseball players and teams. And his New York Times bestselling book, The Signal and the Noise was named by Amazon.com as the number one best non-fiction book of 2012. He's currently the Editor in Chief of the award winning website, FiveThirtyEight and appears on ESPN as an on air commentator. Big round of applause. My pleasure to welcome Nate Silver. >> Thank you. We met backstage. >> Yes. >> It feels weird to re-shake your hand, but you know, for the audience. >> I had to give the intense firm grip. >> Definitely. >> The ninja grip. So you and I have crossed paths kind of digitally in the past, which it really interesting, is I started my career at ESPN. And I started as a production assistant, then later back on air for sports technology. And I go to you to talk about sports because-- >> Yeah. >> Wow, has ESPN upped their game in terms of understanding the importance of data and analytics. And what it brings. Not just to MLB, but across the board. >> No, it's really infused into the way they present the broadcast. You'll have win probability on the bottom line. And they'll incorporate FiveThirtyEight metrics into how they cover college football for example. So, ESPN ... Sports is maybe the perfect, if you're a data scientist, like the perfect kind of test case. And the reason being that sports consists of problems that have rules. And have structure. And when problems have rules and structure, then it's a lot easier to work with. So it's a great way to kind of improve your skills as a data scientist. Of course, there are also important real world problems that are more open ended, and those present different types of challenges. But it's such a natural fit. The teams. Think about the teams playing the World Series tonight. The Dodgers and the Astros are both like very data driven, especially Houston. Golden State Warriors, the NBA Champions, extremely data driven. New England Patriots, relative to an NFL team, it's shifted a little bit, the NFL bar is lower. But the Patriots are certainly very analytical in how they make decisions. So, you can't talk about sports without talking about analytics. >> And I was going to save the baseball question for later. Cause we are moments away from game seven. >> Yeah. >> Is everyone else watching game seven? It's been an incredible series. Probably one of the best of all time. >> Yeah, I mean-- >> You have a prediction here? >> You can mention that too. So I don't have a prediction. FiveThirtyEight has the Dodgers with a 60% chance of winning. >> [Katie] LA Fans. >> So you have two teams that are about equal. But the Dodgers pitching staff is in better shape at the moment. The end of a seven game series. And they're at home. >> But the statistics behind the two teams is pretty incredible. >> Yeah. It's like the first World Series in I think 56 years or something where you have two 100 win teams facing one another. There have been a lot of parity in baseball for a lot of years. Not that many offensive overall juggernauts. But this year, and last year with the Cubs and the Indians too really. But this year, you have really spectacular teams in the World Series. It kind of is a showcase of modern baseball. Lots of home runs. Lots of strikeouts. >> [Katie] Lots of extra innings. >> Lots of extra innings. Good defense. Lots of pitching changes. So if you love the modern baseball game, it's been about the best example that you've had. If you like a little bit more contact, and fewer strikeouts, maybe not so much. But it's been a spectacular and very exciting World Series. It's amazing to talk. MLB is huge with analysis. I mean, hands down. But across the board, if you can provide a few examples. Because there's so many teams in front offices putting such an, just a heavy intensity on the analysis side. And where the teams are going. And if you could provide any specific examples of teams that have really blown your mind. Especially over the last year or two. Because every year it gets more exciting if you will. I mean, so a big thing in baseball is defensive shifts. So if you watch tonight, you'll probably see a couple of plays where if you're used to watching baseball, a guy makes really solid contact. And there's a fielder there that you don't think should be there. But that's really very data driven where you analyze where's this guy hit the ball. That part's not so hard. But also there's game theory involved. Because you have to adjust for the fact that he knows where you're positioning the defenders. He's trying therefore to make adjustments to his own swing and so that's been a major innovation in how baseball is played. You know, how bullpens are used too. Where teams have realized that actually having a guy, across all sports pretty much, realizing the importance of rest. And of fatigue. And that you can be the best pitcher in the world, but guess what? After four or five innings, you're probably not as good as a guy who has a fresh arm necessarily. So I mean, it really is like, these are not subtle things anymore. It's not just oh, on base percentage is valuable. It really effects kind of every strategic decision in baseball. The NBA, if you watch an NBA game tonight, see how many three point shots are taken. That's in part because of data. And teams realizing hey, three points is worth more than two, once you're more than about five feet from the basket, the shooting percentage gets really flat. And so it's revolutionary, right? Like teams that will shoot almost half their shots from the three point range nowadays. Larry Bird, who wound up being one of the greatest three point shooters of all time, took only eight three pointers his first year in the NBA. It's quite noticeable if you watch baseball or basketball in particular. >> Not to focus too much on sports. One final question. In terms of Major League Soccer, and now in NFL, we're having the analysis and having wearables where it can now showcase if they wanted to on screen, heart rate and breathing and how much exertion. How much data is too much data? And when does it ruin the sport? >> So, I don't think, I mean, again, it goes sport by sport a little bit. I think in basketball you actually have a more exciting game. I think the game is more open now. You have more three pointers. You have guys getting higher assist totals. But you know, I don't know. I'm not one of those people who thinks look, if you love baseball or basketball, and you go in to work for the Astros, the Yankees or the Knicks, they probably need some help, right? You really have to be passionate about that sport. Because it's all based on what questions am I asking? As I'm a fan or I guess an employee of the team. Or a player watching the game. And there isn't really any substitute I don't think for the insight and intuition that a curious human has to kind of ask the right questions. So we can talk at great length about what tools do you then apply when you have those questions, but that still comes from people. I don't think machine learning could help with what questions do I want to ask of the data. It might help you get the answers. >> If you have a mid-fielder in a soccer game though, not exerting, only 80%, and you're seeing that on a screen as a fan, and you're saying could that person get fired at the end of the day? One day, with the data? >> So we found that actually some in soccer in particular, some of the better players are actually more still. So Leo Messi, maybe the best player in the world, doesn't move as much as other soccer players do. And the reason being that A) he kind of knows how to position himself in the first place. B) he realizes that you make a run, and you're out of position. That's quite fatiguing. And particularly soccer, like basketball, is a sport where it's incredibly fatiguing. And so, sometimes the guys who conserve their energy, that kind of old school mentality, you have to hustle at every moment. That is not helpful to the team if you're hustling on an irrelevant play. And therefore, on a critical play, can't get back on defense, for example. >> Sports, but also data is moving exponentially as we're just speaking about today. Tech, healthcare, every different industry. Is there any particular that's a favorite of yours to cover? And I imagine they're all different as well. >> I mean, I do like sports. We cover a lot of politics too. Which is different. I mean in politics I think people aren't intuitively as data driven as they might be in sports for example. It's impressive to follow the breakthroughs in artificial intelligence. It started out just as kind of playing games and playing chess and poker and Go and things like that. But you really have seen a lot of breakthroughs in the last couple of years. But yeah, it's kind of infused into everything really. >> You're known for your work in politics though. Especially presidential campaigns. >> Yeah. >> This year, in particular. Was it insanely challenging? What was the most notable thing that came out of any of your predictions? >> I mean, in some ways, looking at the polling was the easiest lens to look at it. So I think there's kind of a myth that last year's result was a big shock and it wasn't really. If you did the modeling in the right way, then you realized that number one, polls have a margin of error. And so when a candidate has a three point lead, that's not particularly safe. Number two, the outcome between different states is correlated. Meaning that it's not that much of a surprise that Clinton lost Wisconsin and Michigan and Pennsylvania and Ohio. You know I'm from Michigan. Have friends from all those states. Kind of the same types of people in those states. Those outcomes are all correlated. So what people thought was a big upset for the polls I think was an example of how data science done carefully and correctly where you understand probabilities, understand correlations. Our model gave Trump a 30% chance of winning. Others models gave him a 1% chance. And so that was interesting in that it showed that number one, that modeling strategies and skill do matter quite a lot. When you have someone saying 30% versus 1%. I mean, that's a very very big spread. And number two, that these aren't like solved problems necessarily. Although again, the problem with elections is that you only have one election every four years. So I can be very confident that I have a better model. Even one year of data doesn't really prove very much. Even five or 10 years doesn't really prove very much. And so, being aware of the limitations to some extent intrinsically in elections when you only get one kind of new training example every four years, there's not really any way around that. There are ways to be more robust to sparce data environments. But if you're identifying different types of business problems to solve, figuring out what's a solvable problem where I can add value with data science is a really key part of what you're doing. >> You're such a leader in this space. In data and analysis. It would be interesting to kind of peek back the curtain, understand how you operate but also how large is your team? How you're putting together information. How quickly you're putting it out. Cause I think in this right now world where everybody wants things instantly-- >> Yeah. >> There's also, you want to be first too in the world of journalism. But you don't want to be inaccurate because that's your credibility. >> We talked about this before, right? I think on average, speed is a little bit overrated in journalism. >> [Katie] I think it's a big problem in journalism. >> Yeah. >> Especially in the tech world. You have to be first. You have to be first. And it's just pumping out, pumping out. And there's got to be more time spent on stories if I can speak subjectively. >> Yeah, for sure. But at the same time, we are reacting to the news. And so we have people that come in, we hire most of our people actually from journalism. >> [Katie] How many people do you have on your team? >> About 35. But, if you get someone who comes in from an academic track for example, they might be surprised at how fast journalism is. That even though we might be slower than the average website, the fact that there's a tragic event in New York, are there things we have to say about that? A candidate drops out of the presidential race, are things we have to say about that. In periods ranging from minutes to days as opposed to kind of weeks to months to years in the academic world. The corporate world moves faster. What is a little different about journalism is that you are expected to have more precision where people notice when you make a mistake. In corporations, you have maybe less transparency. If you make 10 investments and seven of them turn out well, then you'll get a lot of profit from that, right? In journalism, it's a little different. If you make kind of seven predictions or say seven things, and seven of them are very accurate and three of them aren't, you'll still get criticized a lot for the three. Just because that's kind of the way that journalism is. And so the kind of combination of needing, not having that much tolerance for mistakes, but also needing to be fast. That is tricky. And I criticize other journalists sometimes including for not being data driven enough, but the best excuse any journalist has, this is happening really fast and it's my job to kind of figure out in real time what's going on and provide useful information to the readers. And that's really difficult. Especially in a world where literally, I'll probably get off the stage and check my phone and who knows what President Trump will have tweeted or what things will have happened. But it really is a kind of 24/7. >> Well because it's 24/7 with FiveThirtyEight, one of the most well known sites for data, are you feeling micromanagey on your people? Because you do have to hit this balance. You can't have something come out four or five days later. >> Yeah, I'm not -- >> Are you overseeing everything? >> I'm not by nature a micromanager. And so you try to hire well. You try and let people make mistakes. And the flip side of this is that if a news organization that never had any mistakes, never had any corrections, that's raw, right? You have to have some tolerance for error because you are trying to decide things in real time. And figure things out. I think transparency's a big part of that. Say here's what we think, and here's why we think it. If we have a model to say it's not just the final number, here's a lot of detail about how that's calculated. In some case we release the code and the raw data. Sometimes we don't because there's a proprietary advantage. But quite often we're saying we want you to trust us and it's so important that you trust us, here's the model. Go play around with it yourself. Here's the data. And that's also I think an important value. >> That speaks to open source. And your perspective on that in general. >> Yeah, I mean, look, I'm a big fan of open source. I worry that I think sometimes the trends are a little bit away from open source. But by the way, one thing that happens when you share your data or you share your thinking at least in lieu of the data, and you can definitely do both is that readers will catch embarrassing mistakes that you made. By the way, even having open sourceness within your team, I mean we have editors and copy editors who often save you from really embarrassing mistakes. And by the way, it's not necessarily people who have a training in data science. I would guess that of our 35 people, maybe only five to 10 have a kind of formal background in what you would call data science. >> [Katie] I think that speaks to the theme here. >> Yeah. >> [Katie] That everybody's kind of got to be data literate. >> But yeah, it is like you have a good intuition. You have a good BS detector basically. And you have a good intuition for hey, this looks a little bit out of line to me. And sometimes that can be based on domain knowledge, right? We have one of our copy editors, she's a big college football fan. And we had an algorithm we released that tries to predict what the human being selection committee will do, and she was like, why is LSU rated so high? Cause I know that LSU sucks this year. And we looked at it, and she was right. There was a bug where it had forgotten to account for their last game where they lost to Troy or something and so -- >> That also speaks to the human element as well. >> It does. In general as a rule, if you're designing a kind of regression based model, it's different in machine learning where you have more, when you kind of build in the tolerance for error. But if you're trying to do something more precise, then so much of it is just debugging. It's saying that looks wrong to me. And I'm going to investigate that. And sometimes it's not wrong. Sometimes your model actually has an insight that you didn't have yourself. But fairly often, it is. And I think kind of what you learn is like, hey if there's something that bothers me, I want to go investigate that now and debug that now. Because the last thing you want is where all of a sudden, the answer you're putting out there in the world hinges on a mistake that you made. Cause you never know if you have so to speak, 1,000 lines of code and they all perform something differently. You never know when you get in a weird edge case where this one decision you made winds up being the difference between your having a good forecast and a bad one. In a defensible position and a indefensible one. So we definitely are quite diligent and careful. But it's also kind of knowing like, hey, where is an approximation good enough and where do I need more precision? Cause you could also drive yourself crazy in the other direction where you know, it doesn't matter if the answer is 91.2 versus 90. And so you can kind of go 91.2, three, four and it's like kind of A) false precision and B) not a good use of your time. So that's where I do still spend a lot of time is thinking about which problems are "solvable" or approachable with data and which ones aren't. And when they're not by the way, you're still allowed to report on them. We are a news organization so we do traditional reporting as well. And then kind of figuring out when do you need precision versus when is being pointed in the right direction good enough? >> I would love to get inside your brain and see how you operate on just like an everyday walking to Walgreens movement. It's like oh, if I cross the street in .2-- >> It's not, I mean-- >> Is it like maddening in there? >> No, not really. I mean, I'm like-- >> This is an honest question. >> If I'm looking for airfares, I'm a little more careful. But no, part of it's like you don't want to waste time on unimportant decisions, right? I will sometimes, if I can't decide what to eat at a restaurant, I'll flip a coin. If the chicken and the pasta both sound really good-- >> That's not high tech Nate. We want better. >> But that's the point, right? It's like both the chicken and the pasta are going to be really darn good, right? So I'm not going to waste my time trying to figure it out. I'm just going to have an arbitrary way to decide. >> Serious and business, how organizations in the last three to five years have just evolved with this data boom. How are you seeing it as from a consultant point of view? Do you think it's an exciting time? Do you think it's a you must act now time? >> I mean, we do know that you definitely see a lot of talent among the younger generation now. That so FiveThirtyEight has been at ESPN for four years now. And man, the quality of the interns we get has improved so much in four years. The quality of the kind of young hires that we make straight out of college has improved so much in four years. So you definitely do see a younger generation for which this is just part of their bloodstream and part of their DNA. And also, particular fields that we're interested in. So we're interested in people who have both a data and a journalism background. We're interested in people who have a visualization and a coding background. A lot of what we do is very much interactive graphics and so forth. And so we do see those skill sets coming into play a lot more. And so the kind of shortage of talent that had I think frankly been a problem for a long time, I'm optimistic based on the young people in our office, it's a little anecdotal but you can tell that there are so many more programs that are kind of teaching students the right set of skills that maybe weren't taught as much a few years ago. >> But when you're seeing these big organizations, ESPN as perfect example, moving more towards data and analytics than ever before. >> Yeah. >> You would say that's obviously true. >> Oh for sure. >> If you're not moving that direction, you're going to fall behind quickly. >> Yeah and the thing is, if you read my book or I guess people have a copy of the book. In some ways it's saying hey, there are lot of ways to screw up when you're using data. And we've built bad models. We've had models that were bad and got good results. Good models that got bad results and everything else. But the point is that the reason to be out in front of the problem is so you give yourself more runway to make errors and mistakes. And to learn kind of what works and what doesn't and which people to put on the problem. I sometimes do worry that a company says oh we need data. And everyone kind of agrees on that now. We need data science. Then they have some big test case. And they have a failure. And they maybe have a failure because they didn't know really how to use it well enough. But learning from that and iterating on that. And so by the time that you're on the third generation of kind of a problem that you're trying to solve, and you're watching everyone else make the mistake that you made five years ago, I mean, that's really powerful. But that doesn't mean that getting invested in it now, getting invested both in technology and the human capital side is important. >> Final question for you as we run out of time. 2018 beyond, what is your biggest project in terms of data gathering that you're working on? >> There's a midterm election coming up. That's a big thing for us. We're also doing a lot of work with NBA data. So for four years now, the NBA has been collecting player tracking data. So they have 3D cameras in every arena. So they can actually kind of quantify for example how fast a fast break is, for example. Or literally where a player is and where the ball is. For every NBA game now for the past four or five years. And there hasn't really been an overall metric of player value that's taken advantage of that. The teams do it. But in the NBA, the teams are a little bit ahead of journalists and analysts. So we're trying to have a really truly next generation stat. It's a lot of data. Sometimes I now more oversee things than I once did myself. And so you're parsing through many, many, many lines of code. But yeah, so we hope to have that out at some point in the next few months. >> Anything you've personally been passionate about that you've wanted to work on and kind of solve? >> I mean, the NBA thing, I am a pretty big basketball fan. >> You can do better than that. Come on, I want something real personal that you're like I got to crunch the numbers. >> You know, we tried to figure out where the best burrito in America was a few years ago. >> I'm going to end it there. >> Okay. >> Nate, thank you so much for joining us. It's been an absolute pleasure. Thank you. >> Cool, thank you. >> I thought we were going to chat World Series, you know. Burritos, important. I want to thank everybody here in our audience. Let's give him a big round of applause. >> [Nate] Thank you everyone. >> Perfect way to end the day. And for a replay of today's program, just head on over to ibm.com/dsforall. I'm Katie Linendoll. And this has been Data Science for All: It's a Whole New Game. Test one, two. One, two, three. Hi guys, I just want to quickly let you know as you're exiting. A few heads up. Downstairs right now there's going to be a meet and greet with Nate. And we're going to be doing that with clients and customers who are interested. So I would recommend before the game starts, and you lose Nate, head on downstairs. And also the gallery is open until eight p.m. with demos and activations. And tomorrow, make sure to come back too. Because we have exciting stuff. I'll be joining you as your host. And we're kicking off at nine a.m. So bye everybody, thank you so much. >> [Announcer] Ladies and gentlemen, thank you for attending this evening's webcast. If you are not attending all cloud and cognitive summit tomorrow, we ask that you recycle your name badge at the registration desk. Thank you. Also, please note there are two exits on the back of the room on either side of the room. Have a good evening. Ladies and gentlemen, the meet and greet will be on stage. Thank you.

>> Announcer: Live from San Francisco, it's theCUBE covering Spark Summit 2017. Brought to you by Databricks. >> Getting close to the end of the day here at Spark Summit, but we saved the best for last I think. I'm pretty sure about that. I'm David Goad, your host here on theCUBE and we now have data scientists from Riot Games, yes, Riot Games. His name is Wesley Kerr. Wesley, thanks for joining us. >> Thanks for having me. >> What's the best money-making game at Riot Games? >> Well we only have one game. We're known for League of Legends. It came out in 2009, it has been growing and well-received by our fans since then. >> And what's your role there? It says data scientist, but what do you really do? >> So we build models to look at things like in game behavior. We build models to actually help players engage with our store and buy our content. We look at different ways we can, just, improve our player experience. >> Alright well let's talk about a little more under the hood, here. How are you deploying Spark in the game? >> So we relied on Databricks for all of our deployment. We do many different clusters. We have about 14 data scientists that work with us, each one is sort of able to manage their own clusters: spin 'em up, tear 'em down, find their data that way and work with it through Databricks. >> So what else will you cover? You had a keynote session this morning, right? >> Yep. >> Give a recap for theCUBE audience of what you talked about. >> So we talked about our efforts in player behavior where we build models and deploy models that are watching chat between players so we evaluate whether or not players are being unsportsmanlike and come up with ways to, sort of, help them curb that behavior and be more sportsmanlike in our game. >> Oh wow, unsportsmanlike. How do you define that? It's if people are being abusive? >> So what we saw was there are about one or two percent of our games that is some form of serious abuse and that comes in term of hate speech, racism, sexism, things that have no place in the game and so we want them to realize that that language is bad and they shouldn't be using it. >> It's all key word driven or are there other behaviors or things that can indicate? >> So right now it's purely based on things said in chat, but we're currently investigating other, sort of, other ways of measuring that behavior and how it occurs in game and how it could influence what people are saying. >> Maybe like tweets coming from The White House? (laughing) >> Okay, so George. >> We should be able to measure that as well. >> So how about those warriors? (laughing) >> No, George did you want to talk a little bit more >> Sure. >> David: about the technical achievements here? When you look at like trying to measure engagement and sort of maybe it sounds like converting high engagement to store purchases, tell us a little more maybe how that works. >> So we look at, we want. Our game is completely free to play. Players can download, play it all the way through and we really try to create a very engaging game that they want to come back and they want to play and then everything they can buy in the store is actually just cosmetics. So we really hope to build content that our players love and are happy to spend money on. As far as... We just really want engagement to be from around players coming back and playing and having a good time and it's less about how to get that high engagement conversion into monetization as we've seen that players who are happy and loving the game are happy to spend their money. >> So tell us more about how you build some of these models like, you know, turning it into not turning it into Spark code, but how do you analyze it and, sort of, what's the database mechanism for, you know, 'cause the storage layer in Spark, you know, is just like the file system? >> Sure, yeah absolutely. So we are a world-wide game. We're played by over 100 million players around the world >> David: Wow. >> And so that data comes flowing in from all around the world into our centralized data warehouse. That data warehouse has gameplay data so we know how you did in game. It also has time series events, so things that occurred in each game. And our game is really session based so players can come play for an hour, that's one game, and then they leave and come back and play again. And so what we're able to do is then, sort of, look at those models and how they did. And I'll give you an example around our content recommendations. So we look at the champions that you've been playing recently to predict which champions you are likely to play next. And that we can actually just query the database, start building our collaborative filtering models on top of it, and then recommend champions that you may not play now, you may be interested in playing, or we may decide to give you a special discount on a champion if we think it'll resonate well with you. >> And in this case, just to be clear, the champions you're talking about are other players, not models? >> It's actually the in-game avatar. So it's the champion that they play. So we have 130 unique champions and each game you choose which champion you want to play and so then that plays out for like. It's much more like a sport than it is like a game. So it's five v five, online competitive. So there are different objectives on the map. You work with your team to complete those objectives and beat the other team. So we like to think of it like basketball, but with magic and in a virtual world. >> And the teams stay together? Or are they constantly recombining? >> They can disband, yeah. Your next game may find nine other people. If you're playing with your friends then you can just keep queuing up with them as well. So the champions that they control there happen to be who you're playing in that game. >> And when you are trying to anticipate champions that someone might play in the future, what are the variables that you're trying to guess and how long did it take you to build those models? >> Yeah, it's a good question. Right now we are able to sort of leverage the power of our user, our players, so we have 100 million. And so what we do and we have in our game there are roles so, for instance, like there's a center in basketball, we have a bot lane. So we have bottom lane support and bottom lane ADC. So a support character is there to make sure that your ADC is able to defeat the other team. And if you play a lot of support, odds are there are other players in the world who play a lot of support too so we find similar players. We find that if they engaged on the same sorts of champions that you play. For instance, I'm a Leona main and so I play her a lot. And if I were to look at what other people played in addition to Leona it could be things like Braum and so then we would recommend Braum as a champion that you should try out that you've maybe not played yet. >> David: Okay. >> So and then what's the data warehouse that you guys use for the ultimate repository of all this? >> All the data flows into a Hive data warehouse, stored in S3. We have two different ways of interacting with it. One, we can run queries against Hive. It tends to be a bit slower for our use cases. And then our data scientists tend to access that all that data through Databricks and Spark. And it runs much quicker for our use cases. >> Do you take what's in S3 and put it into a parquet format to accelerate? >> Sometimes, so we do some of those rewrites. We do a lot of our secondary ETLs where we're just joining across multiple tables and writing back out. We'll optimize those for our Spark use cases and there's writing back, sort of, read from S3, do some transformations, write back to S3. >> And how latency-sensitive is this? Are you guys trying to make decisions as the player moves along in his level or? >> So historically we've been batch. We do- our recommendations are updated weekly so we haven't needed a much higher cadence. But we're moving to a point where I want to see us be able to actually make recommendations on the client and do it immediately after you've finished a game with, say, Leona, here's an offer for Braum. Go check it out, give it a try in your next game. >> So Wesley what would you like to see developed that hasn't been developed yet that would really help in your business specifically? >> So one thing that's really exciting for gaming right now is procedural generation and artificial intelligence. So here there are a lot of opportunities, you've seen some collaborations between Deep Mind and Blizzard where they're learning to play Starcraft. For me, I think there's a similar world where we have a game that has different sorts of mechanics. So we have a large social piece to our game and teamwork is required. And so understanding how we can leverage that and help influence the future of artificial intelligence is something that I want to see us be able to do. >> Did you talk with anybody here at the Spark Summit about that? >> Anyone who would listen. (laughing) So we chatted some with the teams up at Blizzard and Twitch about some of the things they're doing for natural language as well. >> Alright so what was the most useful conversation you had here at the summit? >> The most useful one that I had, I think, was with the Databricks team. So at the end of my keynote, It was kind of serendipitous, I was talking about some work we had done with deep learning and sort of doing hyper parameter searches over our worker nodes, so actually being able to quickly try out many different models. And in the announcement that morning before my keynote, Tim talked about how they actually have deep learning pipelines now and it was based on a conversation we had had so I was very excited to see it come to fruition and now is open source and we can leverage it. >> Awesome, well, we're up against a hard break here. >> Wesley: Okay. >> We're almost at the end of the day. Wesley, it's been a riot talking to you. We really appreciate it and thank you for coming on the show and sharing your knowledge. >> Wesley: You bet, thanks for having me. >> Alright and that's it, we're going to wrap it up today. We have a wrap-up coming up, as a matter of fact, in just a few minutes. My name is David Goad. You're watching theCUBE at Spark Summit. (upbeat music)