>>Yeah, yeah. >>Welcome back. I'm so excited to introduce our next session empowerment through inclusion, reimagining society and technology. This is a topic that's personally very near and dear to my heart. Did you know that there's only 2% of Latinas in technology as a Latina? I know that there's so much more we could do collectively to improve these gaps and diversity. I thought spot diversity is considered a critical element across all levels of the organization. The data shows countless times. A diverse and inclusive workforce ultimately drives innovation better performance and keeps your employees happier. That's why we're passionate about contributing to this conversation and also partnering with organizations that share our mission of improving diversity across our communities. Last beyond, we hosted the session during a breakfast and we packed the whole room. This year, we're bringing the conversation to the forefront to emphasize the importance of diversity and data and share the positive ramifications that it has for your organization. Joining us for this session are thought spots Chief Data Strategy Officer Cindy Housing and Ruhollah Benjamin, associate professor of African American Studies at Princeton University. Thank you, Paola. So many >>of you have journeyed with me for years now on our efforts to improve diversity and inclusion in the data and analytic space. And >>I would say >>over time we cautiously started commiserating, eventually sharing best practices to make ourselves and our companies better. And I do consider it a milestone. Last year, as Paola mentioned that half the room was filled with our male allies. But I remember one of our Panelists, Natalie Longhurst from Vodafone, suggesting that we move it from a side hallway conversation, early morning breakfast to the main stage. And I >>think it was >>Bill Zang from a I G in Japan. Who said Yes, please. Everyone else agreed, but more than a main stage topic, I want to ask you to think about inclusion beyond your role beyond your company toe. How Data and analytics can be used to impact inclusion and equity for the society as a whole. Are we using data to reveal patterns or to perpetuate problems leading Tobias at scale? You are the experts, the change agents, the leaders that can prevent this. I am thrilled to introduce you to the leading authority on this topic, Rou Ha Benjamin, associate professor of African studies at Princeton University and author of Multiple Books. The Latest Race After Technology. Rou ha Welcome. >>Thank you. Thank you so much for having me. I'm thrilled to be in conversation with you today, and I thought I would just kick things off with some opening reflections on this really important session theme. And then we could jump into discussion. So I'd like us to as a starting point, um, wrestle with these buzzwords, empowerment and inclusion so that we can have them be more than kind of big platitudes and really have them reflected in our workplace cultures and the things that we design in the technologies that we put out into the world. And so to do that, I think we have to move beyond techno determinism, and I'll explain what that means in just a minute. Techno determinism comes in two forms. The first, on your left is the idea that technology automation, um, all of these emerging trends are going to harm us, are going to necessarily harm humanity. They're going to take all the jobs they're going to remove human agency. This is what we might call the techno dystopian version of the story and this is what Hollywood loves to sell us in the form of movies like The Matrix or Terminator. The other version on your right is the techno utopian story that technologies automation. The robots as a shorthand, are going to save humanity. They're gonna make everything more efficient, more equitable. And in this case, on the surface, he seemed like opposing narratives right there, telling us different stories. At least they have different endpoints. But when you pull back the screen and look a little bit more closely, you see that they share an underlying logic that technology is in the driver's seat and that human beings that social society can just respond to what's happening. But we don't really have a say in what technologies air designed and so to move beyond techno determinism the notion that technology is in the driver's seat. We have to put the human agents and agencies back into the story, the protagonists, and think carefully about what the human desires worldviews, values, assumptions are that animate the production of technology. And so we have to put the humans behind the screen back into view. And so that's a very first step and when we do that, we see, as was already mentioned, that it's a very homogeneous group right now in terms of who gets the power and the resource is to produce the digital and physical infrastructure that everyone else has to live with. And so, as a first step, we need to think about how to create more participation of those who are working behind the scenes to design technology now to dig a little more a deeper into this, I want to offer a kind of low tech example before we get to the more hi tech ones. So what you see in front of you here is a simple park bench public bench. It's located in Berkeley, California, which is where I went to graduate school and on this particular visit I was living in Boston, and so I was back in California. It was February. It was freezing where I was coming from, and so I wanted to take a few minutes in between meetings to just lay out in the sun and soak in some vitamin D, and I quickly realized, actually, I couldn't lay down on this bench because of the way it had been designed with these arm rests at intermittent intervals. And so here I thought. Okay, the the armrest have, ah functional reason why they're there. I mean, you could literally rest your elbows there or, um, you know, it can create a little bit of privacy of someone sitting there that you don't know. When I was nine months pregnant, it could help me get up and down or for the elderly, the same thing. So it has a lot of functional reasons, but I also thought about the fact that it prevents people who are homeless from sleeping on the bench. And this is the Bay area that we were talking about where, in fact, the tech boom has gone hand in hand with a housing crisis. Those things have grown in tandem. So innovation has grown within equity because we haven't thought carefully about how to address the social context in which technology grows and blossoms. And so I thought, Okay, this crisis is growing in this area, and so perhaps this is a deliberate attempt to make sure that people don't sleep on the benches by the way that they're designed and where the where they're implemented and So this is what we might call structural inequity. By the way something is designed. It has certain effects that exclude or harm different people. And so it may not necessarily be the intense, but that's the effect. And I did a little digging, and I found, in fact, it's a global phenomenon, this thing that architects called hostile architecture. Er, I found single occupancy benches in Helsinki, so only one booty at a time no laying down there. I found caged benches in France. And in this particular town. What's interesting here is that the mayor put these benches out in this little shopping plaza, and within 24 hours the people in the town rallied together and had them removed. So we see here that just because we have, uh, discriminatory design in our public space doesn't mean we have to live with it. We can actually work together to ensure that our public space reflects our better values. But I think my favorite example of all is the meter bench. In this case, this bench is designed with spikes in them, and to get the spikes to retreat into the bench, you have to feed the meter you have to put some coins in, and I think it buys you about 15 or 20 minutes. Then the spikes come back up. And so you'll be happy to know that in this case, this was designed by a German artists to get people to think critically about issues of design, not just the design of physical space but the design of all kinds of things, public policies. And so we can think about how our public life in general is metered, that it serves those that can pay the price and others are excluded or harm, whether we're talking about education or health care. And the meter bench also presents something interesting. For those of us who care about technology, it creates a technical fix for a social problem. In fact, it started out his art. But some municipalities in different parts of the world have actually adopted this in their public spaces in their parks in order to deter so called lawyers from using that space. And so, by a technical fix, we mean something that creates a short term effect, right. It gets people who may want to sleep on it out of sight. They're unable to use it, but it doesn't address the underlying problems that create that need to sleep outside in the first place. And so, in addition to techno determinism, we have to think critically about technical fixes that don't address the underlying issues that technology is meant to solve. And so this is part of a broader issue of discriminatory design, and we can apply the bench metaphor to all kinds of things that we work with or that we create. And the question we really have to continuously ask ourselves is, What values are we building in to the physical and digital infrastructures around us? What are the spikes that we may unwittingly put into place? Or perhaps we didn't create the spikes. Perhaps we started a new job or a new position, and someone hands us something. This is the way things have always been done. So we inherit the spike bench. What is our responsibility when we noticed that it's creating these kinds of harms or exclusions or technical fixes that are bypassing the underlying problem? What is our responsibility? All of this came to a head in the context of financial technologies. I don't know how many of you remember these high profile cases of tech insiders and CEOs who applied for Apple, the Apple card and, in one case, a husband and wife applied and the husband, the husband received a much higher limit almost 20 times the limit as his wife, even though they shared bank accounts, they lived in Common Law State. And so the question. There was not only the fact that the husband was receiving a much better interest rate and the limit, but also that there was no mechanism for the individuals involved to dispute what was happening. They didn't even know what the factors were that they were being judged that was creating this form of discrimination. So in terms of financial technologies, it's not simply the outcome that's the issue. Or that could be discriminatory, but the process that black boxes, all of the decision making that makes it so that consumers and the general public have no way to question it. No way to understand how they're being judged adversely, and so it's the process not only the product that we have to care a lot about. And so the case of the apple cart is part of a much broader phenomenon of, um, racist and sexist robots. This is how the headlines framed it a few years ago, and I was so interested in this framing because there was a first wave of stories that seemed to be shocked at the prospect that technology is not neutral. Then there was a second wave of stories that seemed less surprised. Well, of course, technology inherits its creator's biases. And now I think we've entered a phase of attempts to override and address the default settings of so called racist and sexist robots, for better or worse. And here robots is just a kind of shorthand, that the way people are talking about automation and emerging technologies more broadly. And so as I was encountering these headlines, I was thinking about how these air, not problems simply brought on by machine learning or AI. They're not all brand new, and so I wanted to contribute to the conversation, a kind of larger context and a longer history for us to think carefully about the social dimensions of technology. And so I developed a concept called the New Jim Code, which plays on the phrase Jim Crow, which is the way that the regime of white supremacy and inequality in this country was defined in a previous era, and I wanted us to think about how that legacy continues to haunt the present, how we might be coding bias into emerging technologies and the danger being that we imagine those technologies to be objective. And so this gives us a language to be able to name this phenomenon so that we can address it and change it under this larger umbrella of the new Jim Code are four distinct ways that this phenomenon takes shape from the more obvious engineered inequity. Those were the kinds of inequalities tech mediated inequalities that we can generally see coming. They're kind of obvious. But then we go down the line and we see it becomes harder to detect. It's happening in our own backyards. It's happening around us, and we don't really have a view into the black box, and so it becomes more insidious. And so in the remaining couple minutes, I'm just just going to give you a taste of the last three of these, and then a move towards conclusion that we can start chatting. So when it comes to default discrimination. This is the way that social inequalities become embedded in emerging technologies because designers of these technologies aren't thinking carefully about history and sociology. Ah, great example of this came Thio headlines last fall when it was found that widely used healthcare algorithm affecting millions of patients, um, was discriminating against black patients. And so what's especially important to note here is that this algorithm healthcare algorithm does not explicitly take note of race. That is to say, it is race neutral by using cost to predict healthcare needs. This digital triaging system unwittingly reproduces health disparities because, on average, black people have incurred fewer costs for a variety of reasons, including structural inequality. So in my review of this study by Obermeyer and colleagues, I want to draw attention to how indifference to social reality can be even more harmful than malicious intent. It doesn't have to be the intent of the designers to create this effect, and so we have to look carefully at how indifference is operating and how race neutrality can be a deadly force. When we move on to the next iteration of the new Jim code coded exposure, there's attention because on the one hand, you see this image where the darker skin individual is not being detected by the facial recognition system, right on the camera or on the computer. And so coated exposure names this tension between wanting to be seen and included and recognized, whether it's in facial recognition or in recommendation systems or in tailored advertising. But the opposite of that, the tension is with when you're over included. When you're surveiled when you're to centered. And so we should note that it's not simply in being left out, that's the problem. But it's in being included in harmful ways. And so I want us to think carefully about the rhetoric of inclusion and understand that inclusion is not simply an end point. It's a process, and it is possible to include people in harmful processes. And so we want to ensure that the process is not harmful for it to really be effective. The last iteration of the new Jim Code. That means the the most insidious, let's say, is technologies that are touted as helping US address bias, so they're not simply including people, but they're actively working to address bias. And so in this case, There are a lot of different companies that are using AI to hire, create hiring software and hiring algorithms, including this one higher view. And the idea is that there there's a lot that AI can keep track of that human beings might miss. And so so the software can make data driven talent decisions. After all, the problem of employment discrimination is widespread and well documented. So the logic goes, Wouldn't this be even more reason to outsource decisions to AI? Well, let's think about this carefully. And this is the look of the idea of techno benevolence trying to do good without fully reckoning with what? How technology can reproduce inequalities. So some colleagues of mine at Princeton, um, tested a natural learning processing algorithm and was looking to see whether it exhibited the same, um, tendencies that psychologists have documented among humans. E. And what they found was that in fact, the algorithm associating black names with negative words and white names with pleasant sounding words. And so this particular audit builds on a classic study done around 2003, before all of the emerging technologies were on the scene where two University of Chicago economists sent out thousands of resumes to employers in Boston and Chicago, and all they did was change the names on those resumes. All of the other work history education were the same, and then they waited to see who would get called back. And the applicants, the fictional applicants with white sounding names received 50% more callbacks than the black applicants. So if you're presented with that study, you might be tempted to say, Well, let's let technology handle it since humans are so biased. But my colleagues here in computer science found that this natural language processing algorithm actually reproduced those same associations with black and white names. So, too, with gender coded words and names Amazon learned a couple years ago when its own hiring algorithm was found discriminating against women. Nevertheless, it should be clear by now why technical fixes that claim to bypass human biases are so desirable. If Onley there was a way to slay centuries of racist and sexist demons with a social justice box beyond desirable, more like magical, magical for employers, perhaps looking to streamline the grueling work of recruitment but a curse from any jobseekers, as this headline puts it, your next interview could be with a racist spot, bringing us back to that problem space we started with just a few minutes ago. So it's worth noting that job seekers are already developing ways to subvert the system by trading answers to employers test and creating fake applications as informal audits of their own. In terms of a more collective response, there's a federation of European Trade unions call you and I Global that's developed a charter of digital rights for work, others that touches on automated and a I based decisions to be included in bargaining agreements. And so this is one of many efforts to change their ecosystem to change the context in which technology is being deployed to ensure more protections and more rights for everyday people in the US There's the algorithmic accountability bill that's been presented, and it's one effort to create some more protections around this ubiquity of automated decisions, and I think we should all be calling from more public accountability when it comes to the widespread use of automated decisions. Another development that keeps me somewhat hopeful is that tech workers themselves are increasingly speaking out against the most egregious forms of corporate collusion with state sanctioned racism. And to get a taste of that, I encourage you to check out the hashtag Tech won't build it. Among other statements that they have made and walking out and petitioning their companies. Who one group said, as the people who build the technologies that Microsoft profits from, we refuse to be complicit in terms of education, which is my own ground zero. Um, it's a place where we can we can grow a more historically and socially literate approach to tech design. And this is just one, um, resource that you all can download, Um, by developed by some wonderful colleagues at the Data and Society Research Institute in New York and the goal of this interventionist threefold to develop an intellectual understanding of how structural racism operates and algorithms, social media platforms and technologies, not yet developed and emotional intelligence concerning how to resolve racially stressful situations within organizations, and a commitment to take action to reduce harms to communities of color. And so as a final way to think about why these things are so important, I want to offer a couple last provocations. The first is for us to think a new about what actually is deep learning when it comes to computation. I want to suggest that computational depth when it comes to a I systems without historical or social depth, is actually superficial learning. And so we need to have a much more interdisciplinary, integrated approach to knowledge production and to observing and understanding patterns that don't simply rely on one discipline in order to map reality. The last provocation is this. If, as I suggested at the start, inequity is woven into the very fabric of our society, it's built into the design of our. Our policies are physical infrastructures and now even our digital infrastructures. That means that each twist, coil and code is a chance for us toe. We've new patterns, practices and politics. The vastness of the problems that we're up against will be their undoing. Once we accept that we're pattern makers. So what does that look like? It looks like refusing color blindness as an anecdote to tech media discrimination rather than refusing to see difference. Let's take stock of how the training data and the models that we're creating have these built in decisions from the past that have often been discriminatory. It means actually thinking about the underside of inclusion, which can be targeting. And how do we create a more participatory rather than predatory form of inclusion? And ultimately, it also means owning our own power in these systems so that we can change the patterns of the past. If we're if we inherit a spiked bench, that doesn't mean that we need to continue using it. We can work together to design more just and equitable technologies. So with that, I look forward to our conversation. >>Thank you, Ruth. Ha. That was I expected it to be amazing, as I have been devouring your book in the last few weeks. So I knew that would be impactful. I know we will never think about park benches again. How it's art. And you laid down the gauntlet. Oh, my goodness. That tech won't build it. Well, I would say if the thoughts about team has any saying that we absolutely will build it and will continue toe educate ourselves. So you made a few points that it doesn't matter if it was intentional or not. So unintentional has as big an impact. Um, how do we address that does it just start with awareness building or how do we address that? >>Yeah, so it's important. I mean, it's important. I have good intentions. And so, by saying that intentions are not the end, all be all. It doesn't mean that we're throwing intentions out. But it is saying that there's so many things that happened in the world, happened unwittingly without someone sitting down to to make it good or bad. And so this goes on both ends. The analogy that I often use is if I'm parked outside and I see someone, you know breaking into my car, I don't run out there and say Now, do you feel Do you feel in your heart that you're a thief? Do you intend to be a thief? I don't go and grill their identity or their intention. Thio harm me, but I look at the effect of their actions, and so in terms of art, the teams that we work on, I think one of the things that we can do again is to have a range of perspectives around the table that can think ahead like chess, about how things might play out, but also once we've sort of created something and it's, you know, it's entered into, you know, the world. We need to have, ah, regular audits and check ins to see when it's going off track just because we intended to do good and set it out when it goes sideways, we need mechanisms, formal mechanisms that actually are built into the process that can get it back on track or even remove it entirely if we find And we see that with different products, right that get re called. And so we need that to be formalized rather than putting the burden on the people that are using these things toe have to raise the awareness or have to come to us like with the apple card, Right? To say this thing is not fair. Why don't we have that built into the process to begin with? >>Yeah, so a couple things. So my dad used to say the road to hell is paved with good intentions, so that's >>yes on. In fact, in the book, I say the road to hell is paved with technical fixes. So they're me and your dad are on the same page, >>and I I love your point about bringing different perspectives. And I often say this is why diversity is not just about business benefits. It's your best recipe for for identifying the early biases in the data sets in the way we build things. And yet it's such a thorny problem to address bringing new people in from tech. So in the absence of that, what do we do? Is it the outside review boards? Or do you think regulation is the best bet as you mentioned a >>few? Yeah, yeah, we need really need a combination of things. I mean, we need So on the one hand, we need something like a do no harm, um, ethos. So with that we see in medicine so that it becomes part of the fabric and the culture of organizations that that those values, the social values, have equal or more weight than the other kinds of economic imperatives. Right. So we have toe have a reckoning in house, but we can't leave it to people who are designing and have a vested interest in getting things to market to regulate themselves. We also need independent accountability. So we need a combination of this and going back just to your point about just thinking about like, the diversity on teams. One really cautionary example comes to mind from last fall, when Google's New Pixel four phone was about to come out and it had a kind of facial recognition component to it that you could open the phone and they had been following this research that shows that facial recognition systems don't work as well on darker skin individuals, right? And so they wanted Thio get a head start. They wanted to prevent that, right? So they had good intentions. They didn't want their phone toe block out darker skin, you know, users from from using it. And so what they did was they were trying to diversify their training data so that the system would work better and they hired contract workers, and they told these contract workers to engage black people, tell them to use the phone play with, you know, some kind of app, take a selfie so that their faces would populate that the training set, But they didn't. They did not tell the people what their faces were gonna be used for, so they withheld some information. They didn't tell them. It was being used for the spatial recognition system, and the contract workers went to the media and said Something's not right. Why are we being told? Withhold information? And in fact, they told them, going back to the park bench example. To give people who are homeless $5 gift cards to play with the phone and get their images in this. And so this all came to light and Google withdrew this research and this process because it was so in line with a long history of using marginalized, most vulnerable people and populations to make technologies better when those technologies are likely going toe, harm them in terms of surveillance and other things. And so I think I bring this up here to go back to our question of how the composition of teams might help address this. I think often about who is in that room making that decision about sending, creating this process of the contract workers and who the selfies and so on. Perhaps it was a racially homogeneous group where people didn't want really sensitive to how this could be experienced or seen, but maybe it was a diverse, racially diverse group and perhaps the history of harm when it comes to science and technology. Maybe they didn't have that disciplinary knowledge. And so it could also be a function of what people knew in the room, how they could do that chest in their head and think how this is gonna play out. It's not gonna play out very well. And the last thing is that maybe there was disciplinary diversity. Maybe there was racial ethnic diversity, but maybe the workplace culture made it to those people. Didn't feel like they could speak up right so you could have all the diversity in the world. But if you don't create a context in which people who have those insights feel like they can speak up and be respected and heard, then you're basically sitting on a reservoir of resource is and you're not tapping into it to ensure T to do right by your company. And so it's one of those cautionary tales I think that we can all learn from to try to create an environment where we can elicit those insights from our team and our and our coworkers, >>your point about the culture. This is really inclusion very different from just diversity and thought. Eso I like to end on a hopeful note. A prescriptive note. You have some of the most influential data and analytics leaders and experts attending virtually here. So if you imagine the way we use data and housing is a great example, mortgage lending has not been equitable for African Americans in particular. But if you imagine the right way to use data, what is the future hold when we've gotten better at this? More aware >>of this? Thank you for that question on DSO. You know, there's a few things that come to mind for me one. And I think mortgage environment is really the perfect sort of context in which to think through the the both. The problem where the solutions may lie. One of the most powerful ways I see data being used by different organizations and groups is to shine a light on the past and ongoing inequities. And so oftentimes, when people see the bias, let's say when it came to like the the hiring algorithm or the language out, they see the names associated with negative or positive words that tends toe have, ah, bigger impact because they think well, Wow, The technology is reflecting these biases. It really must be true. Never mind that people might have been raising the issues in other ways before. But I think one of the most powerful ways we can use data and technology is as a mirror onto existing forms of inequality That then can motivate us to try to address those things. The caution is that we cannot just address those once we come to grips with the problem, the solution is not simply going to be a technical solution. And so we have to understand both the promise of data and the limits of data. So when it comes to, let's say, a software program, let's say Ah, hiring algorithm that now is trained toe look for diversity as opposed to homogeneity and say I get hired through one of those algorithms in a new workplace. I can get through the door and be hired. But if nothing else about that workplace has changed and on a day to day basis I'm still experiencing microaggressions. I'm still experiencing all kinds of issues. Then that technology just gave me access to ah harmful environment, you see, and so this is the idea that we can't simply expect the technology to solve all of our problems. We have to do the hard work. And so I would encourage everyone listening to both except the promise of these tools, but really crucially, um, Thio, understand that the rial kinds of changes that we need to make are gonna be messy. They're not gonna be quick fixes. If you think about how long it took our society to create the kinds of inequities that that we now it lived with, we should expect to do our part, do the work and pass the baton. We're not going to magically like Fairy does create a wonderful algorithm that's gonna help us bypass these issues. It can expose them. But then it's up to us to actually do the hard work of changing our social relations are changing the culture of not just our workplaces but our schools. Our healthcare systems are neighborhoods so that they reflect our better values. >>Yeah. Ha. So beautifully said I think all of us are willing to do the hard work. And I like your point about using it is a mirror and thought spot. We like to say a fact driven world is a better world. It can give us that transparency. So on behalf of everyone, thank you so much for your passion for your hard work and for talking to us. >>Thank you, Cindy. Thank you so much for inviting me. Hey, I live back to you. >>Thank you, Cindy and rou ha. For this fascinating exploration of our society and technology, we're just about ready to move on to our final session of the day. So make sure to tune in for this customer case study session with executives from Sienna and Accenture on driving digital transformation with certain AI.

>> From around the globe, it's theCube, with digital coverage of AWS re:Invent 2020. Special coverage sponsored by, AWS Worldwide Public Sector. >> Okay, welcome back to theCube's coverage, of re:Invent 2020 virtual. It's theCube virtual, I'm John Farrow your host, we're here celebrating, the special coverage of public sector with Sandy Carter, vice president of AWS Public Sector Partners. She heads up the partner group within Public Sector, now in multiple for about a year now. Right Sandy, or so? >> Right, you got it, John. >> About a year? Congratulations, welcome back to theCube, >> Thank you. >> for reason- >> Always a pleasure to be here and what an exciting re:Invent right? >> It's been exciting, we've got wall-to-wall coverage, multiple sets, a lot of actions, virtual it's three weeks, we're not in person we have to do it remote this year. So when real life comes back, we'll bring the Cube back. But I want to take a minute to step back, take a minute to explain your role for the folks that are new to theCube virtual and what you're doing over there at Public Sector. Take a moment to introduce yourself to the new viewers. >> Well, welcome. theCube is phenomenal, and of course we love our new virtual re:Invent as well, as John said, my name is Sandy Carter and I'm vice president with our public sector partners group. So what does that mean? That means I get to work with thousands of partners globally covering exciting verticals like, space and healthcare, education, state and local government, federal government, and more. And what I get to do is, to help our partners learn more about AWS so that they can help our customers really be successful in the marketplace. >> What has been the most, exciting thing for you in the job? >> Well, you know, I love, wow, I love everything about it, but I think one of the things I love the most, is how we in Public Sector, really make technology have a meaningful impact on the world. So John, I get to work with partners like Orbis which is a non-profit they're fighting preventable blindness. They're a partner of ours. They've got something called CyberSec AI which enables us to use machine learning over 20 different machine learning algorithms to detect common eye diseases in seconds. So, you know, that purpose for me is so important. We also work with a partner called Twist Inc it's hard to say, but it just does a phenomenal job with AWS IoT and helps make water pumps, smart pumps. So they are in 7,300 remote locations around the world helping us with clean water. So for me that's probably the most exciting and meaningful part of the job that I have today. >> And it's so impactful because you guys really knew Amazon's business model has always been about enablement from startups to now up and running Public Sector; entities, agencies, education, healthcare, again, and even in spaces, this IoT in space. But you've been on the 100 partner tour over a 100 days. What did you learn, what are you hearing from partners now? What's the messages that you're hearing? >> Well, first of all, it was so exciting. I had a 100 different partner meetings in a 100 days because John, just like you, I missed going around the world and meeting in person. So I said, well, if I can't meet in person I will do a virtual tour and I talked to partners, in 68 different countries. So a couple of things I heard, one is a lot of love for our map program and that's our migration acceleration program. We now have funding available for partners as they assess migration, we can mobilize it and as they migrate it. And you may or may not know, but we have over twice the number of migration competency partners doing business in Public Sector this year, than we did last year. The second thing we heard was that, partners really love our marketing programs. We had some really nice success this year showcasing value for our customers with cyber security. And I love that because security is so important. Andy Jassy always talks about how her customers really have that as priority zeros. So we were able to work with a couple of different areas that we were very proud at and I loved that the partners were too. We did some repeatable solutions with our consulting partners. And then I think the third big takeaway that I saw was just our partners love the AWS technology. I heard a lot about AI and ML. We offered this new program called The Rapid Adoption Assistance Program. It's going global in 2021, and so we help partners brainstorm and envision what they could do with it. And then of course, 5G. 5G is ushering in, kind of a new era of new demand. And we going to to do a PartnerCast on all about 5G for partners in the first quarter. >> Okay, I'm going to put you on the spot. What are the three most talked about programs that you heard? >> Oh, wow, let's see. The three most talked about programs that I heard about, the first one was, is something I'm really excited about. It's called a Think Big for Small Business. It really focuses in on diverse partner groups and types. What it does is it provides just a little bit of extra boost to our small and medium businesses to help them get some of the benefits of our AWS partner program. So companies like MFT they're based down in South Africa it's a husband and wife team that focus on that Black Economic Empowerment rating and they use the program to get some of the go to market capability. So that's number one. Let's see, you said three. Okay, so number two would be our ProServe ready pilot. This helps to accelerate our partner activation and enablement and provides partners a way to get badged on the ProServe best practices get trained up and does opportunity matching. And I think a lot of partners were kind of buzzing about that program and wanting to know more about it. And then ,last but not least, the one that I think of probably really has impact to time to compliance it's called ATO or Authority to Operate and what we do is we help our partners, both technology partners and consulting partners get support for compliance framework. So FedRAMP, of course, we have over 129 solutions right now that are FedRAMPed but we also added John, PCI for financial HIPPA for healthcare, for public safety, IRS 1075 for international GDPR and of course for defense, aisle four, five and six, and CMMC. That program is amazing because it cuts the time to market and have cuts across and have and really steps partners through all of our best practices. I think those are the top three. >> Yeah, I've been like a broken record for the folks that don't know all my interviews I've done with Public Sector over the years. The last one is interesting and I think that's a secret sauce that you guys have done, the compliance piece, being an entrepreneur and starting companies that first three steps in a cloud of dust momentum the flywheel to get going. It's always the hardest and getting the certification if you don't have the resources, it's time consuming. I think you guys really cracked the code on that. I really want to call that out 'cause that's I think really super valuable for the folks that pay attention to and of course sales enablement through the program. So great stuff. Now, given that's all cool, (hands claps) the question I have and I hear all the time is, okay, I'm involved I got a lot of pressure pandemic has forced me to rethink I don't have a lot of IT I don't have a big budget I always complaint but not anymore. Mandate is move fast, get built out, leverage the cloud. Okay, I want to get going. What's the best ways for me to grow with Public Sector? How do I do that if I'm a customer, I really want to... I won't say take a shortcut because there's probably no shortage. How do I throttle up? Quickly, what's your take on that? >> Well, John, first I want to give one star that came to us from a Twilio. They had interviewed a ton of companies and they found that there was more digital transformation since March since when the pandemic started to now than in the last five years. So that just blew me away. And I know all of our partners are looking to see how they can really grow based on that. So if you're a consulting partner, one of the things that we say to help you grow is we've already done some integrations and if you can take advantage of those that can speed up your time to market. So I know know this one, the VMware Cloud on AWS. what a powerful integration, it provides protection of skillsets to your customer, increases your time to market because now VMware, vSphere, VSAN is all on AWS. So it's the same user interface and it really helps to reduce costs. And there's another integration that I think really helps which is Amazon connect one of our fastest growing areas because it's a ML AI, breads solution to help with call centers. It's been integrated with Salesforce but the Service Cloud and the Sales Cloud. So how powerful is that this integrated customer workflow? So I think both of those are really interesting for our consulting partners. >> That's a great point. In fact, well, that's the big part of the story here at re:Invent. These three weeks has been the integration. Salesforce as you mentioned connect has been huge and partner- >> Huge >> so just just great success again, I've seen great momentum. People are seeing their jobs being saved, they're saving lives. People are pretty excited and it's certainly a lot of work you've done in healthcare and education two big areas of activity which is really hard corporation, really, really hard. So congratulations on that and great work. Great to see you, I going to ask you one final question. What's the big message for your customers watching as they prepare for 2021 real life is coming back vaccines on the horizon. We're hearing some good news a lot of great cloud help there. What's your message to send to 2021? >> 2021, for our partners for 2021, one, there is a tremendous growth ahead and tremendous value that our partners have added. And that's both on the mission side, which both Theresa and I discussed during our sessions as well as technology. So I think first messages is, there's lots of growth ahead and a lot of ways that we can add value. Second is, all of those programs and initiatives, there's so much help out there for partners. So look for how you could really accelerate using some of those areas on your customer journey as you're going along. And then finally, I just want John, everybody to know , that we love our partners and AWS is there to help you every step of the way. And if you need anything at all obviously reach out to your PDM or your account manager or you're always welcome to reach out to me. And my final message is just, thank you, through so many different things that have happened in 2020, our partners have come through amazingly with passion with value and just with persistence, never stopping. So thank you to all of our partners out there who've really added so much value to our customers. >> And Amazon is recognizing the leadership of partners in the work you're doing. Your leadership session was awesome for the folks who missed it, check it out on demand. Thank you very much, Sandy for coming on the sharing the update. >> Thank you, John, and great to see all your partners out there. >> Okay, this is theCube virtual covering AWS re:Invent 2020 virtual three weeks, wall-to-wall coverage. A lot of videos ,check out all the videos on demand the leadership sessions, theCube videos and of course the Public Sector video on demand. Micro-site with theCube. I'm John Furrier, thanks for watching. (upbeat music)

>>Thank you. Thank you so much for having me. I'm thrilled to be in conversation with you today. And I thought I would just kick things off with some opening reflections on this really important session theme, and then we can jump into discussion. So I'd like us to, as a starting point, um, wrestle with these buzz words, empowerment and inclusion so that we can, um, have them be more than kind of big platitudes and really have them reflected in our workplace cultures and the things that we design and the technologies that we put out into the world. And so to do that, I think we have to move beyond techno determinism and I'll explain what that means in just a minute. And techno determinism comes in two forms. The first on your left is the idea that technology automate. Um, all of these emerging trends are going to harm us are going to necessarily, um, harm humanity. >>They're going to take all the jobs they're going to remove human agency. This is what we might call the techno dystopian version of the story. And this is what Hollywood loves to sell us in the form of movies like the matrix or Terminator. The other version on your right is the techno utopian story that technologies automation, the robots, as a shorthand are going to save humanity. They're going to make everything more efficient, more equitable. And in this case, on the surface, they seem like opposing narratives, right? They're telling us different stories. At least they have different endpoints, but when you pull back the screen and look a little bit more closely, you see that they share an underlying logic, that technology is in the driver's seat and that human beings, that social society can just respond to what's happening. But we don't, I really have a say in what technologies are designed. >>And so to move beyond techno determinism, the notion that technology is in the driver's seat, we have to put the human agents and agencies back into the story protagonists and think carefully about what the human desires, worldviews values assumptions are that animate the production of technology. We have to put the humans behind the screen back into view. And so that's a very first step in when we do that. We see as was already mentioned that it's a very homogenous group right now in terms of who gets the power and the resources to produce the digital and physical infrastructure that everyone else has to live with. And so, as a first step, we need to think about how to, to create more participation of those who are working behind the scenes to design technology. Now, to dig a little more deeper into this, I want to offer a kind of low tech example before we get to the more high tech ones. >>So what you see in front of you here is a simple park bench public it's located in Berkeley, California, which is where I went to graduate school. And on this one particular visit, I was living in Boston. And so I was back in California, it was February, it was freezing where I was coming from. And so I wanted to take a few minutes in between meetings to just lay out in the sun and soak in some vitamin D. And I quickly realized actually I couldn't lay down on the bench because of the way it had been designed with these arm rests at intermittent intervals. And so here I thought, okay, th th the armrests have a functional reason why they're there. I mean, you could literally rest your elbows there, or, um, you know, it can create a little bit of privacy of someone sitting there that you don't know. >>Um, when I was nine months pregnant, it could help me get up and down or for the elderly the same thing. So it has a lot of functional reasons, but I also thought about the fact that it prevents people who are, are homeless from sleeping on the bench. And this is the Bay area that we're talking about, where in fact, the tech boom has gone hand in hand with a housing crisis. Those things have grown in tandem. So innovation has grown with inequity because we have, I haven't thought carefully about how to address the social context in which technology grows and blossoms. And so I thought, okay, this crisis is growing in this area. And so perhaps this is a deliberate attempt to make sure that people don't sleep on the benches by the way that they're designed and where the, where they're implemented. And so this is what we might call structural inequity, by the way something is designed. >>It has certain yeah. Affects that exclude or harm different people. And so it may not necessarily be the intent, but that's the effect. And I did a little digging and I found, in fact, it's a global phenomenon, this thing that architect next call, hostile architecture around single occupancy, benches and Helsinki. So only one booty at a time, no Nolan down there. I've found caged benches in France. Yeah. And in this particular town, what's interesting here is that the mayor put these benches out in this little shopping Plaza and within 24 hours, the people in the town rally together and have them removed. So we see here that just because we, we have a discriminatory design in our public space, doesn't mean we have to live with it. We can actually work together to ensure that our public space reflects our better values. But I think my favorite example of all is the metered bench. >>And then this case, this bench is designed with spikes in them and to get the spikes to retreat into the bench, you have to feed the meter. You have to put some coins in, and I think it buys you about 15, 20 minutes, then the spikes come back up. And so you will be happy to know that in this case, uh, this was designed by a German artist to get people to think critically about issues of design, not the design of physical space, but the design of all kinds of things, public policies. And so we can think about how our public life in general is metered, that it serves those that can pay the price and others are excluded or harmed. Whether we're talking about education or healthcare. And the meter bench also presents something interesting for those of us who care about technology, it creates a technical fix for a social problem. >>In fact, it started out as art, but some municipalities in different parts of the world have actually adopted this in their public spaces, in their parks in order to deter so-called loiters from using that space. And so by a technical fix, we mean something that creates a short-term effect, right? It gets people who may want to sleep on it out of sight. They're unable to use it, but it doesn't address the underlying problems that create that need to sleep outside of the first place. And so, in addition to techno determinism, we have to think critically about technical fixes, that don't address the underlying issues that the tech tech technology is meant to solve. And so this is part of a broader issue of discriminatory design, and we can apply the bench metaphor to all kinds of things that we work with, or that we create. >>And the question we really have to continuously ask ourselves is what values are we building in to the physical and digital infrastructures around us? What are the spikes that we may unwittingly put into place? Or perhaps we didn't create the spikes. Perhaps we started a new job or a new position, and someone hands us something, this is the way things have always been done. So we inherit the spiked bench. What is our responsibility? When we notice that it's creating these kinds of harms or exclusions or technical fixes that are bypassing the underlying problem, what is our responsibility? All of this came to a head in the context of financial technologies. I don't know how many of you remember these high profile cases of tech insiders and CEOs who applied for apples, >>The Apple card. And in one case, a husband and wife applied, and the husband, the husband received a much higher limit, almost 20 times the limit as his, >>His wife, even though they shared bank accounts, they lived in common law state. Yeah. >>And so the question there was not only the fact that >>The husband was receiving a much better rate and a high and a better >>The interest rate and the limit, but also that there was no mechanism for the individuals involved to dispute what was happening. They didn't even know how, what the factors were that they were being judged that was creating this form of discrimination. So >>In terms of financial technologies, it's not simply the outcome, that's the issue, or that can be discriminatory, >>But the process that black box is all of the decision-making that makes it so that consumers and the general public have no way to question it, no way to understand how they're being judged adversely. And so it's the process, not only the product that we have to care a lot about. And so the case of the Apple card is part of a much broader phenomenon >>Of, um, races >>And sexist robots. This is how the headlines framed it a few years ago. And I was so interested in this framing because there was a first wave of stories that seemed to be shocked at the prospect, that technology is not neutral. Then there was a second wave of stories that seemed less surprised. Well, of course, technology inherits its creators biases. And now I think we've entered a phase of attempts to override and address the default settings of so-called racist and sexist robots for better or worse than here. Robots is just a kind of shorthand that the way that people are talking about automation and emerging technologies more broadly. And so, as I was encountering these headlines, I was thinking about how these are not problems simply brought on by machine learning or AI. They're not all brand new. And so I wanted to contribute to the conversation, a kind of larger context and a longer history for us to think carefully about the social dimensions of technology. And so I developed a concept called the new Jim code, >>Which plays on the phrase, >>Jim Crow, which is the way that the regime of white supremacy and inequality in this country was defined in a previous era. And I wanted us to think about how that legacy continues to haunt the present, how we might be coding bias into emerging technologies and the danger being that we imagine those technologies to be objective. And so this gives us a language to be able to name this phenomenon so that we can address it and change it under this larger umbrella of the new Jim code are four distinct ways that this phenomenon takes shape from the more obvious engineered inequity. Those are the kinds of inequalities tech mediated in the qualities that we can generally see coming. They're kind of obvious, but then we go down the line and we see it becomes harder to detect it's happening in our own backyards, it's happening around us. And we don't really have a view into the black box. And so it becomes more insidious. And so in the remaining couple of minutes, I'm just, just going to give you a taste of the last three of these, and then a move towards conclusion. Then we can start chatting. So when it comes to default discrimination, this is the way that social inequalities >>Become embedded in emerging technologies because designers of these technologies, aren't thinking carefully about history and sociology. A great example of this, uh, came to, um, uh, the headlines last fall when it was found that widely used healthcare algorithm, effecting millions of patients, um, was discriminating against black patients. And so what's especially important to note here is that this algorithm, healthcare algorithm does not explicitly take note of race. That is to say it is race neutral by using cost to predict healthcare needs this digital triaging system unwittingly reproduces health disparities, because on average, black people have incurred fewer costs for a variety of reasons, including structural inequality. So in my review of this study, by Obermeyer and colleagues, I want to draw attention to how indifference to social reality can be even more harmful than malicious intent. It doesn't have to be the intent of the designers to create this effect. >>And so we have to look carefully at how indifference is operating and how race neutrality can be a deadly force. When we move on to the next iteration of the new Jim code, coded exposure, there's a tension because on the one hand, you see this image where the darker skin individual is not being detected by the facial recognition system, right on the camera, on the computer. And so coded exposure names, this tension between wanting to be seen and included and recognized whether it's in facial recognition or in recommendation systems or in tailored advertising. But the opposite of that, the tension is with when you're over, it >>Included when you're surveilled, when you're >>Too centered. And so we should note that it's not simply in being left out, that's the problem, but it's in being included in harmful ways. And so I want us to think carefully about the rhetoric of inclusion and understand that inclusion is not simply an end point, it's a process, and it is possible to include people in harmful processes. And so we want to ensure that the process is not harmful for it to really be effective. The last iteration of the new Jim code. That means the, the most insidious let's say is technologies that are touted as helping us address bias. So they're not simply including people, but they're actively working to address bias. And so in this case, there are a lot of different companies that are using AI to hire, uh, create hiring, um, software and hiring algorithms, including this one higher view. >>And the idea is that there there's a lot that, um, AI can keep track of that human beings might miss. And so, so the software can make data-driven talent decisions after all the problem of employment discrimination is widespread and well-documented, so the logic goes, wouldn't this be even more reason to outsource decisions to AI? Well, let's think about this carefully. And this is the idea of techno benevolence, trying to do good without fully reckoning with what, how technology can reproduce inequalities. So some colleagues of mine at Princeton, um, tested a natural learning processing algorithm and was looking to see whether it exhibited the same, um, tendencies that psychologists have documented among humans. And what they found was that in fact, the algorithm associated black names with negative words and white names with pleasant sounding words. And so this particular audit builds on a classic study done around 2003 before all of the emerging technologies were on the scene where two university of Chicago economists sent out thousands of resumes to employers in Boston and Chicago. >>And all they did was change the names on those resumes. All of the other work history education were the same. And then they waited to see who would get called back and the applicants, the fictional applicants with white sounding names received 50% more callbacks than the, the black applicants. So if you're presented with that study, you might be tempted to say, well, let's let technology handle it since humans are so biased. But my colleagues here in computer science found that this natural language processing algorithm actually reproduced those same associations with black and white names. So two with gender coded words and names as Amazon learned a couple years ago, when its own hiring algorithm was found discriminating against women, nevertheless, it should be clear by now why technical fixes that claim to bypass human biases are so desirable. If only there was a way to slay centuries of racist and sexist demons with a social justice bot beyond desirable, more like magical, magical for employers, perhaps looking to streamline the grueling work of recruitment, but a curse from any job seekers as this headline puts it. >>Your next interview could be with a racist bot, bringing us back to that problem space. We started with just a few minutes ago. So it's worth noting that job seekers are already developing ways to subvert the system by trading answers to employers tests and creating fake applications as informal audits of their own. In terms of a more collective response. There's a Federation of European trade unions call you and I global that's developed a charter of digital rights for workers that touches on automated and AI based decisions to be included in bargaining agreements. And so this is one of many efforts to change the ecosystem, to change the context in which technology is being deployed to ensure more protections and more rights for everyday people in the U S there's the algorithmic accountability bill that's been presented. And it's one effort to create some more protections around this ubiquity of automated decisions. >>And I think we should all be calling for more public accountability when it comes to the widespread use of automated decisions. Another development that keeps me somewhat hopeful is that tech workers themselves are increasingly speaking out against the most egregious forms of corporate collusion with state sanctioned racism. And to get a taste of that, I encourage you to check out the hashtag tech, won't build it among other statements that they've made and walking out and petitioning their companies. One group said as the, at Google at Microsoft wrote as the people who build the technologies that Microsoft profits from, we refuse to be complicit in terms of education, which is my own ground zero. Um, it's a place where we can, we can grow a more historically and socially literate approach to tech design. And this is just one resource that you all can download, um, by developed by some wonderful colleagues at the data and society research Institute in New York. >>And the, the goal of this intervention is threefold to develop an intellectual understanding of how structural racism operates and algorithms, social media platforms and technologies not yet developed and emotional intelligence concerning how to resolve racially stressful situations within organizations and a commitment to take action, to reduce harms to communities of color. And so as a final way to think about why these things are so important, I want to offer, uh, a couple last provocations. The first is pressed to think a new about what actually is deep learning when it comes to computation. I want to suggest that computational depth when it comes to AI systems without historical or social depth is actually superficial learning. And so we need to have a much more interdisciplinary, integrated approach to knowledge production and to observing and understanding patterns that don't simply rely on one discipline in order to map reality. >>The last provocation is this. If as I suggested at the start in the inequity is woven into the very fabric of our society. It's built into the design of our, our policies, our physical infrastructures, and now even our digital infrastructures. That means that each twist coil and code is a chance for us to weave new patterns, practices, and politics. The vastness of the problems that we're up against will be their undoing. Once we, that we are pattern makers. So what does that look like? It looks like refusing colorblindness as an anecdote to tech media discrimination, rather than refusing to see difference. Let's take stock of how the training data and the models that we're creating. Have these built in decisions from the past that have often been discriminatory. It means actually thinking about the underside of inclusion, which can be targeting and how do we create a more participatory rather than predatory form of inclusion. And ultimately it also means owning our own power in these systems so that we can change the patterns of the past. If we're, if we inherit a spiked bench, that doesn't mean that we need to continue using it. We can work together to design more, just an equitable technologies. So with that, I look forward to our conversation.

>> live from Boston, Massachusetts. It's the Cube covering AWS reinforce 2019. Brought to you by Amazon Web service is and its ecosystem partners. >> Okay, welcome back. It was two cubes. Live coverage in Boston, Massachusetts, for Amazon Web services reinforces A W s, his first inaugural conference around security, cloud security and all the benefits of security vendors of bringing. We're here with a man who runs the marketplace and more. Dave McCann Cube, alumni vice president of migration, marketplace and control surfaces. That's a new tail you were that you have here since the last time we talked. Lots changed. Give us the update. Welcome to the Cube. >> Great to be back, ma'am. Believe it's seven months of every event. >> Feels like this. Seven years. You know, you've got a lot new things happening. >> We do >> explain. You have new responsibility. You got the marketplace, which we talked about a great product solutions. What else do you have? >> So we've obviously been expanding our service portfolio, right? So either us is launching. New service is all the time. We have a set of service is a road in the migration of software. So I run. No, the immigration Service's team and interesting. We were sitting in Boston, and that's actually headquartered 800 yards down the road. So there's a set of surfaces around the tools to help you as a CEO. Move your applications onto the clothes. Marketplace is obviously where we want you to find short where you need to buy. And then once you get into the topic of governance, we had one product called Service Catalog and reinvent. We announced a new product. That was a preview called Control. Yesterday we went to G A full availability off control, Terror and Control term service catalog together are in the government space, but we're calling them control service is because it's around controlling the access off teams to particular resources. So that's control service. >> What people moving into the cloud and give us a sense of the the workload. I know you see everything but any patterns that you can see a >> lot of patterns and merging and migration, and they are very industry specific. But there are some common patterns, so you know we're doing migrations and frozen companies were weighed and professional service is run by. Todd Weatherby is engaged in hundreds of those migrations. But we also have no over 70 partners that we've certified of migration partners. Migration partners are doing three times as many migrations as our old professional service is. Team are doing so in collection. There's a lot going on there, one of the common patterns. First of all, everybody is moved a Web development other websites have done. They're all running on the AWS know what they're doing is they're modernizing new applications. So the building in Europe or bring enough over moving onto containers. So it was a lie that ran on a sever server on. As they move into the clothes, they're gonna reshape the throw away. Some of the court brief the court up into micro service is on. Deploy out, Let's see on E. C s, which is continuing. There's a lot of application organization, and then on the migration side, we're seeing applications clearly were migrating a lost a lot of ASAP. So the big partners like Deloitte and Accenture are doing a C P migrations, and we've done a lot of ASAP migrations. And then there are other business applications are being moved with particular software vendors. You know there's a company here in Boston called Pegasystems. They do a world leading workflow platform. We've worked with Pagan, and we have migrated loss of paga warped floors in dozens of paying customers up on the float. >> You innovated on the marketplace, which is where people buy so they can contract with software. So now you got moving to the cloud, buying on the cloud, consuming the cloud and then governing it and managing that aspect all under one cohesive unit. That's you. Is that good? >> Yeah, it's a good way to think about it. It's a san of engineering teams with Coleman purpose for the customer. So you know, one of the things we do AWS is we innovate a lot, and then we organize the engineering teams around a common customer needs. So we said, above all of the computer stories service is on. We pay attention to the application layer. We described the application, So if you think of a migration service is says, I've actually got a service called Discovery, I crawl over your servers and I find what you have way. Then what we do is we have a tool that says, Are you gonna bring and move the till. So you have to build a business case. We just bought a company in Canada called TSA Logic. They had a Super Two for building a business case that said, what would this absolutely running with either of us. >> So is the need of the business case. What's the courtney that you guys have focused on? What was that? >> So, interestingly, we run more Windows Server and the clothes when Microsoft. So you actually have to business keys here. So many windows servers are running on print. What does it look like when a run on either the U. S. And T s so logic? Really good, too. And we find our customers using it. That says, Here's your own prim Windows server configuration with an app on run the mortal What would it look like when it runs on AWS? >> But why would you just do that with a spreadsheet? What? What is the T s so logic do that you couldn't do especially >> well? First of all, you want to make a simple too Somebody has to go run a spreadsheet. They've turned it into a tool that a business years Ercan used a sales person you could use on. They've built on top of a database. So it's got a rich set of choices. You are richer than you put in. A special with a U IE is intuitive, and you're gonna learn it in 20 minutes. I'm not gonna have you made up >> this date in their best practice things like that that you can draw a library >> of what's going down, and it keeps the data store of all the ones we've done. So we're turning that into two. Were giving Old Toller solution architect. >> Well, you got a good thing going on with the marketplace. Good to see you wrapping around those needs there. I gotta ask for the marketplace. Just give us the latest stats. How many subscriptions air in the marketplace these days? What's the overall number in the marketplace? It's >> pretty exciting. Way decided just at San Francisco to announce that we now have over 1,000,000 active subscriptions in the marketplace, which is a main boggling number on its own 1,000,000 subscriptions. Ice of Scrape. Within those subscriptions, we've got over 240 foes and active accounts, you know, and the audience doors you could be an enterprise with 100 cases and in an enterprise. What we typically see is that there are seven or eight teams that are buying or using software, so we'll have seven or eight accounts that have the right to subscribe. So you could be a one team and you're in another team you're buying B I tools. You're buying security tools. So those accounts on what? We're announcing the show for the first time ever. Its security is we have over 100,000 security subscriptions. That's a while. That's a big number. Some companies only have 100 customers, and the market, please. Our customers are switched on 100,000 security. So >> many product listings is that roughly it's just security security. At 300 >> there's over 100 listings. Thing is a product with a price okay on a vendor could be Let's see Paolo off networks or crowdstrike or trains or semantic or McAfee or a brand new company like Twist located of Israel. These companies might have one offer or 20 offers, so we have over 800 offers from over 300. Vendors were having new vendors every week. >> That's the next question. How many security app developers are eyes? Do you have over 300? 300? Okay. About 100. Anyway, I heard >> this morning from Gartner that they believe that are over 1000 security vendors. So I'm only 30% done. I got a little work >> tonight. How >> do you >> govern all this stuff? I was a customer. Sort of Make sure that they're in compliance. >> Great question. Steven Smith yesterday was talking about governance once she moved things on the clothes. It's very elastic. You could be running it today, not running a tomato, running it in I d running in Sydney. So it's easy to fire up running everywhere. So how did the governance team of a company nor watch running where you know, you get into tagging, everything has to be tagged. Everything has to have a cord attached to it. And then you do want to control who gets to use what I may have bought about a cuter appliance. But I don't know that I gave you rates to use it, right, so we could have border on behalf of the company. But I need to grant you access. So we launched a couple of years ago. Service catalog is our first governance to and yesterday we went into full release over new to call the control tower. >> Right. What you announced way reinvents >> preview. And yesterday we went to Jenny. What control does is it Natural Owes me to set up a set of accounts. So if you think of it, your development team, you've got David Kay and tested and the product ain't your brand new to the company. I'm a little worried. What, you're going to get up. You >> don't want to give him the keys to the kingdom, >> so I'm actually going to grant you access to a set of resources, and then I'm gonna apply some rules, or what we call God reels is your brand. You you haven't read my manual, you're in the company. So I'm gonna put a set of God reels on you to make sure that you follow our guide length >> Just training. And so is pressing the wrong button, that kind of thing. So I gotta ask you I mean, on the buying side consumption. I heard you say in a talk upstairs on Monday. You have a buyer, buyer, lead, engineering teams and cellar Let engineering, which tells me that you got a lot of innovation going on the marketplace. So the results are obviously they mention the listings. But one of the trends that's here security conference and it was proper is ecosystems importance in monetization. So back in the old days, Channel partners were a big part of the old computer industry. You're essentially going direct with service listings, which is great. How does that help the channel? Is there sinking around channel as a buyer opportunity? How do you How does that work with the market? Is what your thinking around the relationship between the scale of a simplicity and efficiency, the marketplace with the relationships the channel partners may have with their customers? And how do you bridge that together? What's the thinking >> you've overstayed? Been around a long time? >> Uh, so you have 90 Sydney? Well, the channels have been modernizes the nineties. You think about a >> long time. It's really interesting when we conceived Market please candidly. Way didn't put the channel in marketplace, and in retrospect, that was a miss. Our customers are big customers or small customers. Trust some of the resellers. Some resellers operates surely on price. Some resellers bring a lot of knowledge, even the biggest of the global 2000 Fortune 100. They have a prepared advisor. Let's take a company record. You often got 700 security engineers that are blue chip companies in America trusts or they buy the software the adoptive recommends. So mark it, please really didn't accommodate for Let's Pick another One in Europe, it would be computer center. So in the last two years we've dedicated the data separate engineering team were actually opened up. A team in a different city on their sole customer is a reseller. And so we launch this thing called Consulting Partner Private offer. And so now you're Palo. Also, for your trained, you can authorize active or serious or s h I to be the re sailor at this corporation, and they can actually negotiate the price, which is what a role resellers do. They negotiate price in terms, so we've actually true reseller >> write software for fulfillment through the marketplace. Four partners which are now customers to you now so that they could wrap service is because that's something we talk to. People in the Channel number one conversation is we love the cloud. But how do I make money and that is Service is right. They all want to wrap Service's around, So okay, you guys are delivering this. Is that my getting that right? You guys are riding a direct link in tow marketplace for partners, and they could wrap service is around there, >> will you? Seeing two things? First of all, yes. We're lowering the resale of to sell the software for absolutely. So you re sailor, you can quote software you build rebuild for you so that I become the billing partner for a serious or a billing partner for active on active can use marketplace to fulfill clothes software for their customers. Dan Burns to see you about pretty happy. You crossed the line into a second scenario, which is condone burns attached. Service is on. Clearly, that's a use case we hear usually would we hear use cases way end up through feeling that a little, little not a use case I have enabled, but we've done >> what you're working on It. We've had what the customer. How does the reseller get into the marketplace? What kind of requirements are there. Is it? Is it different than some of your other partners, or is it sort of a similar framework? >> They have to become an approved resale or so First of all, they have to be in a peon partner. I mean, we work tightly with a p N e p M screens partners for AWS. So Josh Hoffman's team Terry Wise, his team, whole part of team screen. The reseller we would only work with resellers are screened and approved by the PM Wants the AP en approved way have no set up a dedicated program team. They work with a reseller with trained them what's involved. Ultimately, however, the relationship is between Splunk in a tree sailor, a five and a three sailor named after a tree sailor or Paulo trend or Croat straight. So it's up to the I S V to tail us that hey, computer centers my reseller. I don't control that relationship. A fulfillment agent you crow strike to save resellers, and I simply have to meet that work so that I get the end customer happy. >> So your enabler in that instance, that's really no, I'm >> really an engine, even team for everybody engineer for the Iast way, engineer for the buyer. And they have to engineer for the re. So >> you have your hands in a lot of the action because you're in the middle of all this marketplace and you must do a lot of planning. I gotta ask you the question and this comes up. That kind of put on my learning all the Amazon lingo covering reinvent for eight years and covering all the different events. So you gotta raise the bar, which is an internal. You keep innovating. Andy Jassy always sucks about removing the undifferentiated heavy lifting. So what is the undifferentiated heavy lifting that you're working toe automate for your customers? >> Great questions. Right now there's probably three. We'll see what the buyer friction is, and then we'll talk about what the sale of friction is. The buyer frustration that is, undifferentiated. Heavy lifting is the interestingly, it's the team process around choosing software. So a couple of customers were on stage yesterday right on those big institutions talked about security software. But in order for an institution to buy that software, there are five groups involved. Security director is choosing the vendor, but procurement has to be involved. Andre. No procurement. We can't be left out the bit. So yesterday we did. The integration to Cooper is a procurement system. So that friction is by subscribing marketplace tied round. Match it with appeal because the p O is what goes on the ledgers with the company. A purchase order. So that has to be a match in purchase order for the marketplace subscription. And then engineers don't Tidwell engineers to always remember you didn't tag it. Hi, this finance nowhere being spent. So we're doing work on working service catalog to do more tagging. And so the buyer wants good tagging procurement integrated. So we're working on a walk slow between marketplace service catalog for procurement. >> Tiring. So you've kind of eliminated procurement or are eliminating procurement as a potential blocker, they use another. Actually, we won't be >> apart for leading procurement. VPs want their V piece of engineering to be happy. >> This is legal. Next. Actually, Greek question. We actually tackled >> legal. First, we did something called Enterprise Code tracked and our customer advisory board Two years ago, one of our buyers, one of our customers, said we're gonna be 100 vendors to deploy it. We're not doing 100 tracks. We've only got one lawyer, You know, 6000 engineers and one lawyer. Well, lawyers, good cord is quickly. So we've created a standard contract. It take stain to persuade legal cause at risk. So we've got a whole bunch of corporations adopting enterprise contract, and we're up to over 75 companies adopting enterprise contract. But legal is apartment >> so modernizing the procurement, a key goal >> procurement, legal, security, engineering. And then the next one is I t finance. So if you think of our budgets on their course teams on AWS, everything needs to be can become visible in either of US budgets. And everything has become visible in course exporter. So we have to call the rate tags. >> I heard a stat that 6,000,000 After moving to the cloud in the next 6,000,000 3 to 5 years, security as a focus reinforces not a summit. It's branded as a W s reinforce, just like reinvents. Same kind of five year for security. What's your impression of the show so far? No, you've been highly active speaking, doing briefing started a customer's burn, the midnight oil with partners and customers What's that? What's your vibe of the show? What's your takeaway? What's the most important thing happening here? What's your what's your summary? >> So I always think you get the truth in the booth. Cut to the chase. I made a customer last night from a major media company who we all know who's in Los Angeles. His comment was weeks, either. These expectations wasn't she wanted to come because he goes to reinvent. Why am I coming to Boston in June? Because I'm gonna go to reinvent November on this. The rates of security for a major media company last night basically said, I love the love. The subject matter, right? It's so security centric. He actually ended up bringing a bunch of people from his team on, and he loves the topics in the stations. The other thing he loved was everybody. Here is insecurity, reinvent. There's lots of people from what's the functions, But everybody here is a security professional. So that was the director of security for a media company. He was at an event talking to one of the suppliers, the marketplace. I asked this president of a very well known security vendor and I said. So what's your reaction to reinforce? And he said, Frankly, when you guys told me it was coming, we didn't really want the bother. It's the end of the quarter. It's a busy time of year. It's another event, he said. I am sure glad we came on. He was standing talking to these VP of marketing, saying, We want to bring more people, make sure, So he's overjoyed. His His comment was, when I go to Rio event 50,000 people but only 5% of their own security. I can't reinforce everybody's insecurity >> in Houston in 2020. Any inside US tow? Why Houston? I have no clue what I actually think >> is really smart about the Vineyard, and this is what a customer said Last night. I met a customer from Connecticut who isn't a load to travel far. They don't get to go to reinvent in Vegas. I think what we did when we came to Boston way tapped into all the states that could drive. So there are people here who don't get to go to reinvent. I think when we go to Houston, we're going to get a whole bunch of takes its customers. Yeah, you don't get a flight to Vegas. So I think it's really good for the customer that people who don't get budget to travel >> makes sense on dry kind of a geographic beograd. The world >> if we're expanding the customers that can learn. So from an education point of view, we're just increase the audience that we're teaching. Great, >> Dave. Great to have you on. Thanks for the insights and congratulations on the new responsibility as you get more coz and around marketplace been very successful. 1,000,000 subscriptions. That's good stuff again. They were >> you reinvented and >> a couple of months, Seven days? What? We're excited. I love covering the growth of the clouds. Certainly cloud security of his own conference. Dave McCann, Vice president Marketplace Migration and Control Service is controlled cattle up. How they how you how you move contract and governed applications in the future. All gonna be happening online. Cloud Mr. Q coverage from Boston. They just reinforced. We right back with more after this short break