(upbeat techno music) >> From Union Square in downtown San Francisco, it's theCUBE, covering PagerDuty Summit '18. Now, here's Jeff Frick. >> Hey welcome back everybody, Jeff Frick here with theCUBE, we're in downtown San Francisco, actually the Westin St. Francis on Union Square, historical property, this beautiful ballroom, lots of brocade and fancy stuff from another era but we're talking about this era and the era of information. We're here at PagerDuty Summit and we're really excited to have one of the keynote speakers, John Allspaw join us. He is the co-founder of Adaptive Capacity Labs. John, great job on the keynote. >> Thanks, thanks a lot, I'm glad that it landed. >> You know it's funny, we go to literally hundreds of tech conferences a year and so often, the tech is talked about but as you brought up, where is the human factors? Where are the people? Where in all these lovely diagrams, as you pointed out, with beautiful lines and everything is very straight and boxes are very clear, that's not really how the real world works at all. >> No, no, yeah, that's what I find really fascinating which is that, certainly get a lot of attention to incidents when they show up, outages and that sort of thing but we don't get a real shot at understanding how non incidents happen and they happen all the time, right? Outages are being prevented all day long. But it doesn't really get our attention. >> Right. >> And it doesn't get our attention because that seems normal and that's the sort of, this assumption that there's a like a quiet sort of background and that an incident is sort of like a punctuation of something bad and that otherwise sticks up, you know, like Mount Hood, right? >> Right, right. >> But the fact of the matter is there's so much going on. >> Right. >> And that's actually that stuff that's going on, is this activity that people are doing to prevent outages continually and that's what I find fascinating. >> So you really never get like your classic kind of experiment where you can isolate the variables, right? >> No. >> Because they're all completely co-mingled, all the time? >> Yeah, and that's what fascinating. What I like and we always say is that we study cognitive work, and the difference between sort of these types of human factors and cognitive work studies and the difference between that and say sort of classic psychology is classic psychology can be done in a lab. We study cognition in the wild. >> Right, right. >> As they say. The natural laboratory that is the world. >> Right, and the other thing I thought you brought up which was really interesting is really kind of what's the point, right? Is the point just to fix it? Is the point to try to identify this little link and fix it? Or is the point kind of a higher level objective which is to actually learn so that we're doing the things in the future that keep this thing from happening again? And you summarized it really, really well and you talked about the post mortem which you said, "Are you doing this report to be read or are you doing it to be filed?" Very different objectives, going to have a very different report at the end of the process. >> Right, right, right, yeah. I think that the sort of the danger is if we, as an industry, I think we just need to bring some attention to that and the good news is that it's hard work to look at incidents in a different way. It's a way that we're not used to. It's effectively qualitative research. It's difficult but it's not impossible, it can be learned, it can be taught and my hope is that sort of these sorts of bringing attention to the topics will get people to be curious and want to understand more. >> Right and really take it up a notch and I think, again, you have some really easy to implement lessons there, like what are the questions? Document the questions, >> Yeah. >> In the post mortem. >> Yeah. >> Document the concerns in the post mortem. Did those concerns happen? Did they not happen? Why didn't they happen? So really kind of take it up a level from the incident, really, as kind of a catalyst for a conversation and learning but that's really not what the foundational effort should be around, is fixing that little thing? >> Right, right. Well and that's the thing, is if the goal is to fix, and that is the goal, you're going to find something to fix. It may or may not be helpful. What you fix comes from exploring and there are things that shouldn't be fixed, right now. Everybody's making decisions, I mean, this is the entire premise of Agile which is that continual iterative re prioritization, recalibration of what's important so we'll be happy to put effort into that but yet, it seems disingenuous to phone it in. >> Right. >> When it comes to understanding incidents. >> Right, right. You got on to so many things, we could go forever and ever. One of things you talked about and it's often spoke about, is winners write the history books. It was really about the bias that you bring to a problem. What do you think is the most important and what filter and lens are you both looking at the problem, reporting the problem or diagnosing and then reporting the problem which may or may not be root cause, may or may not be the most important thing about that but those biases influences not only is that problem perceived but then documented, resolved and talked about after the fact. Really important. >> Yeah, yeah, absolutely and there's something really paradoxical about that. One of the things that it brings to mind is that I don't think that yet we are in a world where we, when I say we, I mean the software industry, will bring attention to a report on near misses. The scenarios where, you know what? You thought you were in dev but you were in prod and you ran a command that if it had a couple of other parameters, it would have destroyed everything but it turns out that actually, it was this one, you know these couple of characters made it such that it was a near miss. It wasn't a big deal. Is that an incident, right? >> Right, right. >> On the one hand you could say, well there are no customer impact. >> Right. >> So therefore let me look up on my, oh, no, that's not an incident so therefore we shouldn't pay any attention to it. But think of any other sort of high tempo, high consequence domain? >> Right. >> They've learned, aviation is a good example. There are organizations in aviation that will, actually and they find them to be incredibly useful because they're low risk things to pay attention to. It didn't happen this time but we can bring attention to the possibility that it might go poorly the next time. >> Right, so what triggers the action to recognize that you had a near miss? And is that working it's way into best practices dev ops? >> Well, I mean, at my organization, at Etsy, I certainly, full disclosure, I made quite a good number of mistakes at Etsy. This isn't one of them. Getting into habit of what had happened there was people sending PSA e-mails, public service announcements and it was basically the format was, hey everybody, check this out, I was doing this and I went to go do blah, and I almost exploded everybody. So FYI if you're doing this, don't do this. Everything's cool and I'm going to put in these things to sort of help it out, but until we get that done, be really careful about this part, you know, whatever. Even just that, even small things like that, keep the topic of how precarious these scenarios can be in the minds of people who aren't experiencing incidents. >> Right, right. >> Tomorrow you might be that one, or tomorrow you might be, and so here's your colleague like taking the time to spend some effort, could be saving your bacon tomorrow. >> Right. >> You might be in the similar spot. >> Right. How's it codified and how is it communicated. So another concept you touched on, which has a broader implication, but you talked about specifically and really that's diversity of opinions leads to better decision making and you gave some examples of bringing in disparate members of various teams with different experiences, points of view. >> Yeah. >> To pull out things like the esoteric knowledge, to pull out the institutional knowledge. >> Yeah. >> But more importantly, to pull out a different point of view. So we hear about it a lot with diversity of teams, and sects, and culture, et cetera but even with the context of solving an engineering problem diversity and points of view does lead to better problem solving. >> I want to make sort of a crisp clarification. It is the variety of perspectives actually the variety of expertise and the variety of experience, not opinions or perspectives. Perspective you can probably, that's word you can probably go with. I wouldn't say diversity of opinion, that has a connotation that is not concrete enough. >> Okay. >> What we're talking about is cognitive work, how people assess this is something that requires my attention. It requires my attention in these ways based on my experience with this particular type of problem over this different variations of it. >> Right. >> Yeah, I mean the general sense is, but the phrase diversity of opinion generally has like a connotation of the individual attribute of a person. I'm not talking about that. I'm talking about the-- >> These are individual attributes that have been gleaned through experience-- >> It's not an attribute, it's experience. >> It's experience, right okay. >> Right, exactly. An attribute of me is that I'm 5'9", my experience is that I have seen Apache break in a myriad of different, surprising ways, right? (laughter) There's sort of the difference. >> Right, right the difference, okay. But then the other point you brought up even in that conversation was it's always messy, there's always trade-offs, is you know, you get management overhead as soon as you have more than one person working on a problem, right? Now you have communication overhead, you've got management overhead so now you're pulling resources from actually devoting it to the task at hand of trying to solve the problem versus having to devote resources to bring other people up to speed, communicate, et cetera. So it's not even a really easy trade off? >> Oh yeah. >> Or not trade off, I mean but there's consequences to the action. >> Oh yeah, absolutely, absolutely. And again, I think coping with complexity requires an equal amount of complexity, right? You might not say that a baseball team that is very good at doing double plays, right? Which is a pretty hard thing to pull off even in professional baseball. Would you say that the coach represents overhead? I don't know if you would say it that way exactly but there's certainly limitations to the sports metaphor. I like very much a renewed emphasis on building, maintaining and sort of, resolving incidents with software as much more benefiting from collaborative work. >> Right, right. >> Meaning real sort of teamwork. >> Right. >> Not just sort of sparse collaboration. >> Right, right. Well John, it's a fascinating field, we could go on all day long. >> Yes we could. >> Unfortunately, we're going to have to leave it there but really, really enjoyed the conversation. >> Great. >> And also the keynote earlier today. >> Great thanks a lot. Thanks for talking. >> Alright, thank you. He's John, I'm Jeff, you're watching theCUBE. We're at PagerDuty Summit at the Westin St. Francis, Union Square. Thanks for watching. (upbeat techno music)