>>Yeah, yeah. >>Welcome back. I'm so excited to introduce our next session empowerment through inclusion, reimagining society and technology. This is a topic that's personally very near and dear to my heart. Did you know that there's only 2% of Latinas in technology as a Latina? I know that there's so much more we could do collectively to improve these gaps and diversity. I thought spot diversity is considered a critical element across all levels of the organization. The data shows countless times. A diverse and inclusive workforce ultimately drives innovation better performance and keeps your employees happier. That's why we're passionate about contributing to this conversation and also partnering with organizations that share our mission of improving diversity across our communities. Last beyond, we hosted the session during a breakfast and we packed the whole room. This year, we're bringing the conversation to the forefront to emphasize the importance of diversity and data and share the positive ramifications that it has for your organization. Joining us for this session are thought spots Chief Data Strategy Officer Cindy Housing and Ruhollah Benjamin, associate professor of African American Studies at Princeton University. Thank you, Paola. So many >>of you have journeyed with me for years now on our efforts to improve diversity and inclusion in the data and analytic space. And >>I would say >>over time we cautiously started commiserating, eventually sharing best practices to make ourselves and our companies better. And I do consider it a milestone. Last year, as Paola mentioned that half the room was filled with our male allies. But I remember one of our Panelists, Natalie Longhurst from Vodafone, suggesting that we move it from a side hallway conversation, early morning breakfast to the main stage. And I >>think it was >>Bill Zang from a I G in Japan. Who said Yes, please. Everyone else agreed, but more than a main stage topic, I want to ask you to think about inclusion beyond your role beyond your company toe. How Data and analytics can be used to impact inclusion and equity for the society as a whole. Are we using data to reveal patterns or to perpetuate problems leading Tobias at scale? You are the experts, the change agents, the leaders that can prevent this. I am thrilled to introduce you to the leading authority on this topic, Rou Ha Benjamin, associate professor of African studies at Princeton University and author of Multiple Books. The Latest Race After Technology. Rou ha Welcome. >>Thank you. Thank you so much for having me. I'm thrilled to be in conversation with you today, and I thought I would just kick things off with some opening reflections on this really important session theme. And then we could jump into discussion. So I'd like us to as a starting point, um, wrestle with these buzzwords, empowerment and inclusion so that we can have them be more than kind of big platitudes and really have them reflected in our workplace cultures and the things that we design in the technologies that we put out into the world. And so to do that, I think we have to move beyond techno determinism, and I'll explain what that means in just a minute. Techno determinism comes in two forms. The first, on your left is the idea that technology automation, um, all of these emerging trends are going to harm us, are going to necessarily harm humanity. They're going to take all the jobs they're going to remove human agency. This is what we might call the techno dystopian version of the story and this is what Hollywood loves to sell us in the form of movies like The Matrix or Terminator. The other version on your right is the techno utopian story that technologies automation. The robots as a shorthand, are going to save humanity. They're gonna make everything more efficient, more equitable. And in this case, on the surface, he seemed like opposing narratives right there, telling us different stories. At least they have different endpoints. But when you pull back the screen and look a little bit more closely, you see that they share an underlying logic that technology is in the driver's seat and that human beings that social society can just respond to what's happening. But we don't really have a say in what technologies air designed and so to move beyond techno determinism the notion that technology is in the driver's seat. We have to put the human agents and agencies back into the story, the protagonists, and think carefully about what the human desires worldviews, values, assumptions are that animate the production of technology. And so we have to put the humans behind the screen back into view. And so that's a very first step and when we do that, we see, as was already mentioned, that it's a very homogeneous group right now in terms of who gets the power and the resource is to produce the digital and physical infrastructure that everyone else has to live with. And so, as a first step, we need to think about how to create more participation of those who are working behind the scenes to design technology now to dig a little more a deeper into this, I want to offer a kind of low tech example before we get to the more hi tech ones. So what you see in front of you here is a simple park bench public bench. It's located in Berkeley, California, which is where I went to graduate school and on this particular visit I was living in Boston, and so I was back in California. It was February. It was freezing where I was coming from, and so I wanted to take a few minutes in between meetings to just lay out in the sun and soak in some vitamin D, and I quickly realized, actually, I couldn't lay down on this bench because of the way it had been designed with these arm rests at intermittent intervals. And so here I thought. Okay, the the armrest have, ah functional reason why they're there. I mean, you could literally rest your elbows there or, um, you know, it can create a little bit of privacy of someone sitting there that you don't know. When I was nine months pregnant, it could help me get up and down or for the elderly, the same thing. So it has a lot of functional reasons, but I also thought about the fact that it prevents people who are homeless from sleeping on the bench. And this is the Bay area that we were talking about where, in fact, the tech boom has gone hand in hand with a housing crisis. Those things have grown in tandem. So innovation has grown within equity because we haven't thought carefully about how to address the social context in which technology grows and blossoms. And so I thought, Okay, this crisis is growing in this area, and so perhaps this is a deliberate attempt to make sure that people don't sleep on the benches by the way that they're designed and where the where they're implemented and So this is what we might call structural inequity. By the way something is designed. It has certain effects that exclude or harm different people. And so it may not necessarily be the intense, but that's the effect. And I did a little digging, and I found, in fact, it's a global phenomenon, this thing that architects called hostile architecture. Er, I found single occupancy benches in Helsinki, so only one booty at a time no laying down there. I found caged benches in France. And in this particular town. What's interesting here is that the mayor put these benches out in this little shopping plaza, and within 24 hours the people in the town rallied together and had them removed. So we see here that just because we have, uh, discriminatory design in our public space doesn't mean we have to live with it. We can actually work together to ensure that our public space reflects our better values. But I think my favorite example of all is the meter bench. In this case, this bench is designed with spikes in them, and to get the spikes to retreat into the bench, you have to feed the meter you have to put some coins in, and I think it buys you about 15 or 20 minutes. Then the spikes come back up. And so you'll be happy to know that in this case, this was designed by a German artists to get people to think critically about issues of design, not just the design of physical space but the design of all kinds of things, public policies. And so we can think about how our public life in general is metered, that it serves those that can pay the price and others are excluded or harm, whether we're talking about education or health care. And the meter bench also presents something interesting. For those of us who care about technology, it creates a technical fix for a social problem. In fact, it started out his art. But some municipalities in different parts of the world have actually adopted this in their public spaces in their parks in order to deter so called lawyers from using that space. And so, by a technical fix, we mean something that creates a short term effect, right. It gets people who may want to sleep on it out of sight. They're unable to use it, but it doesn't address the underlying problems that create that need to sleep outside in the first place. And so, in addition to techno determinism, we have to think critically about technical fixes that don't address the underlying issues that technology is meant to solve. And so this is part of a broader issue of discriminatory design, and we can apply the bench metaphor to all kinds of things that we work with or that we create. And the question we really have to continuously ask ourselves is, What values are we building in to the physical and digital infrastructures around us? What are the spikes that we may unwittingly put into place? Or perhaps we didn't create the spikes. Perhaps we started a new job or a new position, and someone hands us something. This is the way things have always been done. So we inherit the spike bench. What is our responsibility when we noticed that it's creating these kinds of harms or exclusions or technical fixes that are bypassing the underlying problem? What is our responsibility? All of this came to a head in the context of financial technologies. I don't know how many of you remember these high profile cases of tech insiders and CEOs who applied for Apple, the Apple card and, in one case, a husband and wife applied and the husband, the husband received a much higher limit almost 20 times the limit as his wife, even though they shared bank accounts, they lived in Common Law State. And so the question. There was not only the fact that the husband was receiving a much better interest rate and the limit, but also that there was no mechanism for the individuals involved to dispute what was happening. They didn't even know what the factors were that they were being judged that was creating this form of discrimination. So in terms of financial technologies, it's not simply the outcome that's the issue. Or that could be discriminatory, but the process that black boxes, all of the decision making that makes it so that consumers and the general public have no way to question it. No way to understand how they're being judged adversely, and so it's the process not only the product that we have to care a lot about. And so the case of the apple cart is part of a much broader phenomenon of, um, racist and sexist robots. This is how the headlines framed it a few years ago, and I was so interested in this framing because there was a first wave of stories that seemed to be shocked at the prospect that technology is not neutral. Then there was a second wave of stories that seemed less surprised. Well, of course, technology inherits its creator's biases. And now I think we've entered a phase of attempts to override and address the default settings of so called racist and sexist robots, for better or worse. And here robots is just a kind of shorthand, that the way people are talking about automation and emerging technologies more broadly. And so as I was encountering these headlines, I was thinking about how these air, not problems simply brought on by machine learning or AI. They're not all brand new, and so I wanted to contribute to the conversation, a kind of larger context and a longer history for us to think carefully about the social dimensions of technology. And so I developed a concept called the New Jim Code, which plays on the phrase Jim Crow, which is the way that the regime of white supremacy and inequality in this country was defined in a previous era, and I wanted us to think about how that legacy continues to haunt the present, how we might be coding bias into emerging technologies and the danger being that we imagine those technologies to be objective. And so this gives us a language to be able to name this phenomenon so that we can address it and change it under this larger umbrella of the new Jim Code are four distinct ways that this phenomenon takes shape from the more obvious engineered inequity. Those were the kinds of inequalities tech mediated inequalities that we can generally see coming. They're kind of obvious. But then we go down the line and we see it becomes harder to detect. It's happening in our own backyards. It's happening around us, and we don't really have a view into the black box, and so it becomes more insidious. And so in the remaining couple minutes, I'm just just going to give you a taste of the last three of these, and then a move towards conclusion that we can start chatting. So when it comes to default discrimination. This is the way that social inequalities become embedded in emerging technologies because designers of these technologies aren't thinking carefully about history and sociology. Ah, great example of this came Thio headlines last fall when it was found that widely used healthcare algorithm affecting millions of patients, um, was discriminating against black patients. And so what's especially important to note here is that this algorithm healthcare algorithm does not explicitly take note of race. That is to say, it is race neutral by using cost to predict healthcare needs. This digital triaging system unwittingly reproduces health disparities because, on average, black people have incurred fewer costs for a variety of reasons, including structural inequality. So in my review of this study by Obermeyer and colleagues, I want to draw attention to how indifference to social reality can be even more harmful than malicious intent. It doesn't have to be the intent of the designers to create this effect, and so we have to look carefully at how indifference is operating and how race neutrality can be a deadly force. When we move on to the next iteration of the new Jim code coded exposure, there's attention because on the one hand, you see this image where the darker skin individual is not being detected by the facial recognition system, right on the camera or on the computer. And so coated exposure names this tension between wanting to be seen and included and recognized, whether it's in facial recognition or in recommendation systems or in tailored advertising. But the opposite of that, the tension is with when you're over included. When you're surveiled when you're to centered. And so we should note that it's not simply in being left out, that's the problem. But it's in being included in harmful ways. And so I want us to think carefully about the rhetoric of inclusion and understand that inclusion is not simply an end point. It's a process, and it is possible to include people in harmful processes. And so we want to ensure that the process is not harmful for it to really be effective. The last iteration of the new Jim Code. That means the the most insidious, let's say, is technologies that are touted as helping US address bias, so they're not simply including people, but they're actively working to address bias. And so in this case, There are a lot of different companies that are using AI to hire, create hiring software and hiring algorithms, including this one higher view. And the idea is that there there's a lot that AI can keep track of that human beings might miss. And so so the software can make data driven talent decisions. After all, the problem of employment discrimination is widespread and well documented. So the logic goes, Wouldn't this be even more reason to outsource decisions to AI? Well, let's think about this carefully. And this is the look of the idea of techno benevolence trying to do good without fully reckoning with what? How technology can reproduce inequalities. So some colleagues of mine at Princeton, um, tested a natural learning processing algorithm and was looking to see whether it exhibited the same, um, tendencies that psychologists have documented among humans. E. And what they found was that in fact, the algorithm associating black names with negative words and white names with pleasant sounding words. And so this particular audit builds on a classic study done around 2003, before all of the emerging technologies were on the scene where two University of Chicago economists sent out thousands of resumes to employers in Boston and Chicago, and all they did was change the names on those resumes. All of the other work history education were the same, and then they waited to see who would get called back. And the applicants, the fictional applicants with white sounding names received 50% more callbacks than the black applicants. So if you're presented with that study, you might be tempted to say, Well, let's let technology handle it since humans are so biased. But my colleagues here in computer science found that this natural language processing algorithm actually reproduced those same associations with black and white names. So, too, with gender coded words and names Amazon learned a couple years ago when its own hiring algorithm was found discriminating against women. Nevertheless, it should be clear by now why technical fixes that claim to bypass human biases are so desirable. If Onley there was a way to slay centuries of racist and sexist demons with a social justice box beyond desirable, more like magical, magical for employers, perhaps looking to streamline the grueling work of recruitment but a curse from any jobseekers, as this headline puts it, your next interview could be with a racist spot, bringing us back to that problem space we started with just a few minutes ago. So it's worth noting that job seekers are already developing ways to subvert the system by trading answers to employers test and creating fake applications as informal audits of their own. In terms of a more collective response, there's a federation of European Trade unions call you and I Global that's developed a charter of digital rights for work, others that touches on automated and a I based decisions to be included in bargaining agreements. And so this is one of many efforts to change their ecosystem to change the context in which technology is being deployed to ensure more protections and more rights for everyday people in the US There's the algorithmic accountability bill that's been presented, and it's one effort to create some more protections around this ubiquity of automated decisions, and I think we should all be calling from more public accountability when it comes to the widespread use of automated decisions. Another development that keeps me somewhat hopeful is that tech workers themselves are increasingly speaking out against the most egregious forms of corporate collusion with state sanctioned racism. And to get a taste of that, I encourage you to check out the hashtag Tech won't build it. Among other statements that they have made and walking out and petitioning their companies. Who one group said, as the people who build the technologies that Microsoft profits from, we refuse to be complicit in terms of education, which is my own ground zero. Um, it's a place where we can we can grow a more historically and socially literate approach to tech design. And this is just one, um, resource that you all can download, Um, by developed by some wonderful colleagues at the Data and Society Research Institute in New York and the goal of this interventionist threefold to develop an intellectual understanding of how structural racism operates and algorithms, social media platforms and technologies, not yet developed and emotional intelligence concerning how to resolve racially stressful situations within organizations, and a commitment to take action to reduce harms to communities of color. And so as a final way to think about why these things are so important, I want to offer a couple last provocations. The first is for us to think a new about what actually is deep learning when it comes to computation. I want to suggest that computational depth when it comes to a I systems without historical or social depth, is actually superficial learning. And so we need to have a much more interdisciplinary, integrated approach to knowledge production and to observing and understanding patterns that don't simply rely on one discipline in order to map reality. The last provocation is this. If, as I suggested at the start, inequity is woven into the very fabric of our society, it's built into the design of our. Our policies are physical infrastructures and now even our digital infrastructures. That means that each twist, coil and code is a chance for us toe. We've new patterns, practices and politics. The vastness of the problems that we're up against will be their undoing. Once we accept that we're pattern makers. So what does that look like? It looks like refusing color blindness as an anecdote to tech media discrimination rather than refusing to see difference. Let's take stock of how the training data and the models that we're creating have these built in decisions from the past that have often been discriminatory. It means actually thinking about the underside of inclusion, which can be targeting. And how do we create a more participatory rather than predatory form of inclusion? And ultimately, it also means owning our own power in these systems so that we can change the patterns of the past. If we're if we inherit a spiked bench, that doesn't mean that we need to continue using it. We can work together to design more just and equitable technologies. So with that, I look forward to our conversation. >>Thank you, Ruth. Ha. That was I expected it to be amazing, as I have been devouring your book in the last few weeks. So I knew that would be impactful. I know we will never think about park benches again. How it's art. And you laid down the gauntlet. Oh, my goodness. That tech won't build it. Well, I would say if the thoughts about team has any saying that we absolutely will build it and will continue toe educate ourselves. So you made a few points that it doesn't matter if it was intentional or not. So unintentional has as big an impact. Um, how do we address that does it just start with awareness building or how do we address that? >>Yeah, so it's important. I mean, it's important. I have good intentions. And so, by saying that intentions are not the end, all be all. It doesn't mean that we're throwing intentions out. But it is saying that there's so many things that happened in the world, happened unwittingly without someone sitting down to to make it good or bad. And so this goes on both ends. The analogy that I often use is if I'm parked outside and I see someone, you know breaking into my car, I don't run out there and say Now, do you feel Do you feel in your heart that you're a thief? Do you intend to be a thief? I don't go and grill their identity or their intention. Thio harm me, but I look at the effect of their actions, and so in terms of art, the teams that we work on, I think one of the things that we can do again is to have a range of perspectives around the table that can think ahead like chess, about how things might play out, but also once we've sort of created something and it's, you know, it's entered into, you know, the world. We need to have, ah, regular audits and check ins to see when it's going off track just because we intended to do good and set it out when it goes sideways, we need mechanisms, formal mechanisms that actually are built into the process that can get it back on track or even remove it entirely if we find And we see that with different products, right that get re called. And so we need that to be formalized rather than putting the burden on the people that are using these things toe have to raise the awareness or have to come to us like with the apple card, Right? To say this thing is not fair. Why don't we have that built into the process to begin with? >>Yeah, so a couple things. So my dad used to say the road to hell is paved with good intentions, so that's >>yes on. In fact, in the book, I say the road to hell is paved with technical fixes. So they're me and your dad are on the same page, >>and I I love your point about bringing different perspectives. And I often say this is why diversity is not just about business benefits. It's your best recipe for for identifying the early biases in the data sets in the way we build things. And yet it's such a thorny problem to address bringing new people in from tech. So in the absence of that, what do we do? Is it the outside review boards? Or do you think regulation is the best bet as you mentioned a >>few? Yeah, yeah, we need really need a combination of things. I mean, we need So on the one hand, we need something like a do no harm, um, ethos. So with that we see in medicine so that it becomes part of the fabric and the culture of organizations that that those values, the social values, have equal or more weight than the other kinds of economic imperatives. Right. So we have toe have a reckoning in house, but we can't leave it to people who are designing and have a vested interest in getting things to market to regulate themselves. We also need independent accountability. So we need a combination of this and going back just to your point about just thinking about like, the diversity on teams. One really cautionary example comes to mind from last fall, when Google's New Pixel four phone was about to come out and it had a kind of facial recognition component to it that you could open the phone and they had been following this research that shows that facial recognition systems don't work as well on darker skin individuals, right? And so they wanted Thio get a head start. They wanted to prevent that, right? So they had good intentions. They didn't want their phone toe block out darker skin, you know, users from from using it. And so what they did was they were trying to diversify their training data so that the system would work better and they hired contract workers, and they told these contract workers to engage black people, tell them to use the phone play with, you know, some kind of app, take a selfie so that their faces would populate that the training set, But they didn't. They did not tell the people what their faces were gonna be used for, so they withheld some information. They didn't tell them. It was being used for the spatial recognition system, and the contract workers went to the media and said Something's not right. Why are we being told? Withhold information? And in fact, they told them, going back to the park bench example. To give people who are homeless $5 gift cards to play with the phone and get their images in this. And so this all came to light and Google withdrew this research and this process because it was so in line with a long history of using marginalized, most vulnerable people and populations to make technologies better when those technologies are likely going toe, harm them in terms of surveillance and other things. And so I think I bring this up here to go back to our question of how the composition of teams might help address this. I think often about who is in that room making that decision about sending, creating this process of the contract workers and who the selfies and so on. Perhaps it was a racially homogeneous group where people didn't want really sensitive to how this could be experienced or seen, but maybe it was a diverse, racially diverse group and perhaps the history of harm when it comes to science and technology. Maybe they didn't have that disciplinary knowledge. And so it could also be a function of what people knew in the room, how they could do that chest in their head and think how this is gonna play out. It's not gonna play out very well. And the last thing is that maybe there was disciplinary diversity. Maybe there was racial ethnic diversity, but maybe the workplace culture made it to those people. Didn't feel like they could speak up right so you could have all the diversity in the world. But if you don't create a context in which people who have those insights feel like they can speak up and be respected and heard, then you're basically sitting on a reservoir of resource is and you're not tapping into it to ensure T to do right by your company. And so it's one of those cautionary tales I think that we can all learn from to try to create an environment where we can elicit those insights from our team and our and our coworkers, >>your point about the culture. This is really inclusion very different from just diversity and thought. Eso I like to end on a hopeful note. A prescriptive note. You have some of the most influential data and analytics leaders and experts attending virtually here. So if you imagine the way we use data and housing is a great example, mortgage lending has not been equitable for African Americans in particular. But if you imagine the right way to use data, what is the future hold when we've gotten better at this? More aware >>of this? Thank you for that question on DSO. You know, there's a few things that come to mind for me one. And I think mortgage environment is really the perfect sort of context in which to think through the the both. The problem where the solutions may lie. One of the most powerful ways I see data being used by different organizations and groups is to shine a light on the past and ongoing inequities. And so oftentimes, when people see the bias, let's say when it came to like the the hiring algorithm or the language out, they see the names associated with negative or positive words that tends toe have, ah, bigger impact because they think well, Wow, The technology is reflecting these biases. It really must be true. Never mind that people might have been raising the issues in other ways before. But I think one of the most powerful ways we can use data and technology is as a mirror onto existing forms of inequality That then can motivate us to try to address those things. The caution is that we cannot just address those once we come to grips with the problem, the solution is not simply going to be a technical solution. And so we have to understand both the promise of data and the limits of data. So when it comes to, let's say, a software program, let's say Ah, hiring algorithm that now is trained toe look for diversity as opposed to homogeneity and say I get hired through one of those algorithms in a new workplace. I can get through the door and be hired. But if nothing else about that workplace has changed and on a day to day basis I'm still experiencing microaggressions. I'm still experiencing all kinds of issues. Then that technology just gave me access to ah harmful environment, you see, and so this is the idea that we can't simply expect the technology to solve all of our problems. We have to do the hard work. And so I would encourage everyone listening to both except the promise of these tools, but really crucially, um, Thio, understand that the rial kinds of changes that we need to make are gonna be messy. They're not gonna be quick fixes. If you think about how long it took our society to create the kinds of inequities that that we now it lived with, we should expect to do our part, do the work and pass the baton. We're not going to magically like Fairy does create a wonderful algorithm that's gonna help us bypass these issues. It can expose them. But then it's up to us to actually do the hard work of changing our social relations are changing the culture of not just our workplaces but our schools. Our healthcare systems are neighborhoods so that they reflect our better values. >>Yeah. Ha. So beautifully said I think all of us are willing to do the hard work. And I like your point about using it is a mirror and thought spot. We like to say a fact driven world is a better world. It can give us that transparency. So on behalf of everyone, thank you so much for your passion for your hard work and for talking to us. >>Thank you, Cindy. Thank you so much for inviting me. Hey, I live back to you. >>Thank you, Cindy and rou ha. For this fascinating exploration of our society and technology, we're just about ready to move on to our final session of the day. So make sure to tune in for this customer case study session with executives from Sienna and Accenture on driving digital transformation with certain AI.

>>welcome back. And thanks for joining us for our second session. External data, your new leading indicators. We'll be hearing from industry leaders as they share best practices and challenges in leveraging external data. This panel will be a true conversation on the part of the possible. All right, let's get to >>it >>today. We're excited to be joined by thought spots. Chief Data Strategy Officer Cindy Housing Deloitte's chief data officer Manteo, the founder and CEO of Eagle Alfa. And it Kilduff and Snowflakes, VP of data marketplace and customer product strategy. Matt Glickman. Cindy. Without further ado, the floor is yours. >>Thank you, Mallory. And I am thrilled to have this brilliant team joining us from around the world. And they really bring each a very unique perspective. So I'm going to start from further away. Emmett, Welcome. Where you joining us from? >>Thanks for having us, Cindy. I'm joining from Dublin, Ireland, >>great. And and tell us a little bit about Eagle Alfa. What do you dio >>from a company's perspective? Think of Eagle Alfa as an aggregator off all the external data sets on a word I'll use a few times. Today is a big advantage we could bring companies is we have a data concierge service. There's so much data we can help identify the right data sets depending on the specific needs of the company. >>Yeah. And so, Emma, you know, people think I was a little I kind of shocked the industry. Going from gardener to a tech startup. Um, you have had a brave journey as well, Going from financial services to starting this company, really pioneering it with I think the most data sets of any of thes is that right? >>Yes, it was. It was a big jump to go from Morgan Stanley. Uh, leave the comforts of that environment Thio, PowerPoint deck and myself raising funding eight years ago s So it was a big jump on. We were very early in our market. It's in the last few years where there's been real momentum and adoption by various types of verticals. The hedge funds were first, maybe then private equity, but corporate sar are following quite quickly from behind. That will be the biggest users, in our view, by by a significant distance. >>Yeah, great. Thank um, it So we're going to go a little farther a field now, but back to the U. S. So, Juan, where you joining us from? >>Hey, Cindy. Thanks for having me. I'm joining you from Houston, Texas. >>Great. Used to be my home. Yeah, probably see Rice University back there. And you have a distinct perspective serving both Deloitte customers externally, but also internally. Can you tell us about that? >>Yeah, absolutely. So I serve as the Lord consultants, chief data officer, and as a professional service firm, I have the responsibility for overseeing our overall data agenda, which includes both the way we use data and insights to run and operate our own business, but also in how we develop data and insights services that we then take to market and how we serve our dealers and clients. >>Great. Thank you, Juan. And last but not least, Matt Glickman. Kind of in my own backyard in New York. Right, Matt? >>Correct. Joining I haven't been into the city and many months, but yes, um, based in New York. >>Okay. Great. And so, Matt, you and Emmett also, you know, brave pioneers in this space, and I'm remembering a conversation you and I shared when you were still a J. P. Morgan, I believe. And you're Goldman Sachs. Sorry. Sorry. Goldman. Can you Can you share that with us? >>Sure. I made the move back in 2015. Um, when everyone thought, you know, my wife, my wife included that I was crazy. I don't know if I would call it Comfortable was emitted, but particularly had been there for a long time on git suffered in some ways. A lot of the pains we're talking about today, given the number of data, says that the amount of of new data sets that are always demand for having run analytics teams at Goldman, seeing the pain and realizing that this pain was not unique to Goldman Sachs, it was being replicated everywhere across the industry, um, in a mind boggling way and and the fortuitous, um, luck to have one of snowflakes. Founders come to pitch snowflake to Goldman a little bit early. Um, they became a customer later, but a little bit early in 2014. And, you know, I realized that this was clearly, you know, the answer from first principles on bond. If I ever was going to leave, this was a problem. I was acutely aware of. And I also was aware of how much the man that was in financial services for a better solution and how the cloud could really solve this problem in particular the ability to not have to move data in and out of these organizations. And this was something that I saw the future of. Thank you, Andi, that this was, you know, sort of the pain that people just expected to pay. Um, this price if you need a data, there was method you had thio. You had to use you either ftp data in and out. You had data that was being, you know, dropped off and, you know, maybe in in in a new ways and cloud buckets or a P i s You have to suck all this data down and reconstruct it. And God forbid the formats change. It was, you know, a nightmare. And then having issues with data, you had a what you were seeing internally. You look nothing like what the data vendors were seeing because they want a completely different system, maybe model completely differently. Um, but this was just the way things were. Everyone had firewalls. Everyone had their own data centers. There was no other way on git was super costly. And you know this. I won't even share the the details of you know, the errors that would occur in the pain that would come from that, Um what I realized it was confirmed. What I saw it snowflake at the time was once everyone moves to run their actual workloads in this in the cloud right where you're now beyond your firewall, you'll have all this scale. But on top of that, you'll be able to point at data from these vendors were not there the traditional data vendors. Or, you know, this new wave of alternative data vendors, for example, like the ones that eagle out for brings together And bring these all these data sets together with your own internal data without moving it. Yeah, this was a fundamental shift of what you know, it's in some ways, it was a side effect of everyone moving to the cloud for costs and scale and elasticity. But as a side effect of that is what we talked about, You know it snowflake summit, you know, yesterday was this notion of a data cloud that would connect data between regions between cloud vendors between customers in a way where you could now reference data. Just like your reference websites today, I don't download CNN dot com. I point at it, and it points me to something else. I'm always seeing the latest version, obviously, and we can, you know, all collaborate on what I'm seeing on that website. That's the same thing that now can happen with data. So And I saw this as what was possible, and I distinctly asked the question, you know, the CEO of the time Is this possible? And not only was it possible it was a fundamental construct that was built into the way that snowflake was delivered. And then, lastly, this is what we learned. And I think this is what you know. M It also has been touting is that it's all great if data is out there and even if you lower that bar of access where data doesn't have to move, how do I know? Right? If I'm back to sitting at Goldman Sachs, how do I know what data is available to me now in this this you know, connected data network eso we released our data marketplace, which was a very different kind of marketplace than these of the past. Where for us, it was really like a global catalog that would elect a consumer data consumer. Noah data was available, but also level the playing field. Now we're now, you know, Eagle, Alfa, or even, you know, a new alternative data vendor build something in their in their basement can now publish that data set so that the world could see and consume and be aligned to, you know, snowflakes, core business, and not where we wouldn't have to be competing or having to take, um, any kind of custody of that data. So adding that catalog to this now ubiquitous access, um really changed the game and, you know, and then now I seem like a genius for making this move. But back then, like I said, we've seen I seem like instant. I was insane. >>Well, given, given that snowflake was the hottest aipo like ever, you were a genius. Uh, doing this, you know, six years in advance. E think we all agree on that, But, you know, a lot of this is still visionary. Um, you know, some of the most leading companies are already doing this. But one What? What is your take our Are you best in class customers still moving the data? Or is this like they're at least thinking about data monetization? What are you seeing from your perspective? >>Yeah, I mean, I did you know, the overall appreciation and understanding of you know, one. I got to get my house in order around my data, um, has something that has been, you know, understood and acted upon. Andi, I do agree that there is a shift now that says, you know, data silos alone aren't necessarily gonna bring me, you know, new and unique insights on dso enriching that with external third party data is absolutely, you know, sort of the the ship that we're seeing our customers undergo. Um, what I find extremely interesting in this space and what some of the most mature clients are doing is, you know, really taking advantage of these data marketplaces. But building data partnerships right there from what mutually exclusive, where there is a win win scenario for for you know, that organization and that could be, you know, retail customers or life science customers like with pandemic, right the way we saw companies that weren't naturally sharing information are now building these data partnership right that are going are going into mutually benefit, you know, all organizations that are sort of part of that value to Andi. I think that's the sort of really important criteria. And how we're seeing our clients that are extremely successful at this is that partnership has benefits on both sides of that equation, right? Both the data provider and then the consumer of that. And there has to be, you know, some way to ensure that both parties are are are learning right, gaining you insights to support, you know, whatever their business organization going on. >>Yeah, great one. So those data partnerships getting across the full value chain of sharing data and analytics Emmett, you work on both sides of the equation here, helping companies. Let's say let's say data providers maybe, like, you know, cast with human mobility monetize that. But then also people that are new to it. Where you seeing the top use cases? Well, >>interestingly, I agree with one of the supply side. One of the interesting trends is we're seeing a lot more data coming from large Corporates. Whether they're listed are private equity backed, as opposed to maybe data startups that are earning money just through data monetization. I think that's a great trend. I think that means a lot of the best. Data said it data is yet to come, um, in terms off the tough economy and how that's changed. I think the category that's had the most momentum and your references is Geo location data. It's that was the category at our conference in December 2000 and 12 that was pipped as the category to watch in 2019. On it didn't become that at all. Um, there were some regulatory concerns for certain types of geo data, but with with covert 19, it's Bean absolutely critical for governments, ministries of finance, central banks, municipalities, Thio crunch that data to understand what's happening in a real time basis. But from a company perspective, it's obviously critical as well. In terms of planning when customers might be back in the High Street on DSO, fourth traditionally consumer transaction data of all the 26 categories in our taxonomy has been the most popular. But Geo is definitely catching up your slide. Talked about being a tough economy. Just one point to contradict that for certain pockets of our clients, e commerce companies are having a field day, obviously, on they are very data driven and tech literate on day are they are really good client base for us because they're incredibly hungry, firm or data to help drive various, uh, decision making. >>Yeah, So fair enough. Some sectors of the economy e commerce, electron, ICS, healthcare are doing great. Others travel, hospitality, Um, super challenging. So I like your quote. The best is yet to come, >>but >>that's data sets is yet to come. And I do think the cloud is enabling that because we could get rid of some of the messy manual data flows that Matt you talked about, but nonetheless, Still, one of the hardest things is the data map. Things combining internal and external >>when >>you might not even have good master data. Common keys on your internal data. So any advice for this? Anyone who wants to take that? >>Sure I can. I can I can start. That's okay. I do think you know, one of the first problems is just a cataloging of the information that's out there. Um, you know, at least within our organization. When I took on this role, we were, you know, a large buyer of third party data. But our organization as a whole didn't necessarily have full visibility into what was being bought and for what purpose. And so having a catalog that helps us internally navigate what data we have and how we're gonna use it was sort of step number one. Um, so I think that's absolutely important. Um, I would say if we could go from having that catalog, you know, created manually to more automated to me, that's sort of the next step in our evolution, because everyone is saying right, the ongoing, uh, you know, creation of new external data sets. It's only going to get richer on DSO. We wanna be able to take advantage of that, you know, at the at the pacing speed, that data is being created. So going from Emanuel catalog to anonymous >>data >>catalog, I think, is a key capability for us. But then you know, to your second point, Cindy is how doe I then connect that to our own internal data to drive greater greater insights and how we run our business or how we serve our customers. Andi, that one you know really is a It's a tricky is a tricky, uh, question because I think it just depends on what data we're looking toe leverage. You know, we have this concept just around. Not not all data is created equal. And when you think about governance and you think about the management of your master data, your internal nomenclature on how you define and run your business, you know that that entire ecosystem begins to get extremely massive and it gets very broad and very deep on DSO for us. You know, government and master data management is absolutely important. But we took a very sort of prioritized approach on which domains do we really need to get right that drive the greatest results for our organization on dso mapping those domains like client data or employee data to these external third party data sources across this catalog was really the the unlocked for us versus trying to create this, you know, massive connection between all the external data that we're, uh, leveraging as well as all of our own internal data eso for us. I think it was very. It was a very tailored, prioritized approach to connecting internal data to external data based on the domains that matter most to our business. >>So if the domains so customer important domain and maybe that's looking at things, um, you know, whether it's social media data or customer transactions, you prioritized first by that, Is that right? >>That's correct. That's correct. >>And so, then, Matt, I'm going to throw it back to you because snowflake is in a unique position. You actually get to see what are the most popular data sets is is that playing out what one described are you seeing that play out? >>I I'd say Watch this space. Like like you said. I mean this. We've you know, I think we start with the data club. We solve that that movement problem, which I think was really the barrier that you tended to not even have a chance to focus on this mapping problem. Um, this notion of concordance, I think this is where I see the big next momentum in this space is going to be a flurry of traditional and new startups who deliver this concordance or knowledge graph as a service where this is no longer a problem that I have to solve internal to my organization. The notion of mastering which is again when everyone has to do in every organization like they used to have to do with moving data into the organization goes away. And this becomes like, I find the best of breed for the different scopes of data that I have. And it's delivered to me as a, you know, as a cloud service that just takes my data. My internal data maps it to these 2nd and 3rd party data sets. Um, all delivered to me, you know, a service. >>Yeah, well, that would be brilliant concordance as a service or or clean clean master data as a service. Um, using augmented data prep would be brilliant. So let's hope we get there. Um, you know, so 2020 has been a wild ride for everyone. If I could ask each of you imagine what is the art of the possible or looking ahead to the next to your and that you are you already mentioned the best is yet to come. Can you want to drill down on that. What what part of the best is yet to come or what is your already two possible? >>Just just a brief comment on mapping. Just this week we published a white paper on mapping, which is available for for anyone on eagle alfa dot com. It's It's a massive challenge. It's very difficult to solve. Just with technology Onda people have tried to solve it and get a certain level of accuracy, but can't get to 100% which which, which, which makes it difficult to solve it. If if if there is a new service coming out against 100% I'm all ears and that there will be a massive step forward for the entire data industry, even if it comes in a few years time, let alone next year, I think going back to the comment on data Cindy. Yes, I think boards of companies are Mawr and Mawr. Viewing data as an asset as opposed to an expense are a cost center on bond. They are looking therefore to get their internal house in order, as one was saying, but also monetize the data they are sitting on lots of companies. They're sitting on potentially valuable data. It's not all valuable on a lot of cases. They think it's worth a lot more than it is being frank. But in some cases there is valuable data on bond. If monetized, it can drop to the bottom line on. So I think that bodes well right across the world. A lot of the best date is yet to come on. I think a lot of firms like Deloitte are very well positioned to help drive that adoption because they are the trusted advisor to a lot of these Corporates. Um, so that's one thing. I think, from a company perspective. It's still we're still at the first base. It's quite frustrating how slow a lot of companies are to move and adopt, and some of them are haven't hired CDO. Some of them don't have their internal house in order. I think that has to change next year. I think if we have this conference at this time next year, I would expect that would hopefully be close to the tipping point for Corporates to use external data. And the Malcolm Gladwell tipping point on the final point I make is I think, that will hopefully start to see multi department use as opposed to silos again. Parliaments and silos, hopefully will be more coordinated on the company's side. Data could be used by marketing by sales by r and D by strategy by finance holds external data. So it really, hopefully will be coordinated by this time next year. >>Yeah, Thank you. So, to your point, there recently was an article to about one of the airlines that their data actually has more value than the company itself now. So I know, I know. We're counting on, you know, integrators trusted advisers like Deloitte to help us get there. Uh, one what? What do you think? And if I can also drill down, you know, financial services was early toe all of this because they needed the early signals. And and we talk about, you know, is is external data now more valuable than internal? Because we need those early signals in just such a different economy. >>Yeah, I think you know, for me, it's it's the seamless integration of all these external data sources and and the signals that organizations need and how to bring those into, you know, the day to day operations of your organization, right? So how do you bring those into, You know, you're planning process. How do you bring that into your sales process on DSO? I think for me success or or where I see the that the use and adoption of this is it's got to get down to that level off of operations for organizations. For this to continue to move at the pace and deliver the value that you know, we're all describing. I think we're going to get there. But I think until organizations truly get down to that level of operations and how they're using this data, it'll sort of seem like a Bolton, right? So for me, I think it's all about Mawr, the seamless integration. And I think to what Matt mentioned just around services that could help connect external data with internal data. I'll take that one step beyond and say, How can we have the data connect itself? Eso I had references Thio, you know, automation and machine learning. Um, there's significant advances in terms of how we're seeing, you know, mapping to occur in a auto generated fashion. I think this specific space and again the connection between external and internal data is a prime example of where we need to disrupt that, you know, sort of traditional data pipeline on. Try to automate that as much as possible. And let's have the data, you know, connect itself because it then sort of supports. You know, the first concept which waas How do we make it more seamless and integrated into, you know, the business processes of the organization's >>Yeah, great ones. So you two are thinking those automated, more intelligent data pipelines will get us there faster. Matt, you already gave us one. Great, Uh, look ahead, Any more to add to >>it, I'll give you I'll give you two more. One is a bit controversial, but I'll throw that you anyway, um, going back to the point that one made about data partnerships What you were saying Cindy about, you know, the value. These companies, you know, tends to be somehow sometimes more about the data they have than the actual service they provide. I predict you're going to see a wave of mergers and acquisitions. Um, that it's solely about locking down access to data as opposed to having data open up. Um to the broader, you know, economy, if I can, whether that be a retailer or, you know, insurance company was thes prime data assets. Um, you know, they could try to monetize that themselves, But if someone could acquire them and get exclusive access that data, I think that's going to be a wave of, um, in a that is gonna be like, Well, we bought this for this amount of money because of their data assets s. So I think that's gonna be a big wave. And it'll be maybe under the guise of data partnerships. But it really be about, you know, get locking down exclusive access to valuable data as opposed to trying toe monetize it itself number one. And then lastly, you know. Now, did you have this kind of ubiquity of data in this interconnected data network? Well, we're starting to see, and I think going to see a big wave of is hyper personalization of applications where instead of having the application have the data itself Have me Matt at Snowflake. Bring my data graph to applications. Right? This decoupling of we always talk about how you get data out of these applications. It's sort of the reverse was saying Now I want to bring all of my data access that I have 1st, 2nd and 3rd party into my application. Instead of having to think about getting all the data out of these applications, I think about it how when you you know, using a workout app in the consumer space, right? I can connect my Spotify or connect my apple music into that app to personalize the experience and bring my music list to that. Imagine if I could do that, you know, in a in a CRM. Imagine I could do that in a risk management. Imagine I could do that in a marketing app where I can bring my entire data graph with me and personalize that experience for, you know, for given what I have. And I think again, you know, partners like thoughts. But I think in a unique position to help enable that capability, you know, for this next wave of of applications that really take advantage of this decoupling of data. But having data flow into the app tied to me as opposed to having the APP have to know about my data ahead of time, >>Yeah, yeah, So that is very forward thinking. So I'll end with a prediction and a best practice. I am predicting that the organizations that really leverage external data, new data sources, not just whether or what have you and modernize those data flows will outperform the organizations that don't. And as a best practice to getting there, I the CDOs that own this have at least visibility into everything they're purchasing can save millions of dollars in duplicate spend. So, Thio, get their three key takeaways. Identify the leading indicators and market signals The data you need Thio. Better identify that. Consolidate those purchases and please explore the data sets the range of data sets data providers that we have on the thought spot. Atlas Marketplace Mallory over to you. >>Wow. Thank you. That was incredible. Thank you. To all of our Panelists for being here and sharing that wisdom. We really appreciate it. For those of you at home, stay close by. Our third session is coming right up and we'll be joined by our partner AWS and get to see how you can leverage the full power of your data cloud complete with the demo. Make sure to tune in to see you >>then