Working at the Intersection of Technology and Race
A Conversation with Cat Wade
Cat Wade has been affiliated with the Harvard University Edmond J. Safra Center for Ethics and the Embedded EthiCS project, and is currently a PhD candidate in Harvard’s Philosophy department, working on algorithmic discrimination. Her dissertation explores the consequences of our increasing existence in digital lifeworlds. She maintains that objections to digital objects and synthetic media such as digital fashion reveal important and largely unjustifiable biases against the virtual world. Beyond the examination of these biases, she uses synthetic media as a means to examine what the relationship ought to be between our relatively static analog selves and our potentially unbounded digital selves. She argues that while we ought to embrace the leaky potential of our avatars, past and present injustices mean that not everything is on the table for digital manipulation, including race, and in certain cases, age. Finally, she explores what the ramifications of these insights are for algorithmic discrimination, which she examines primarily through the lens of content-curating algorithms.
Susan Carney Lynch: Thank you Cat for taking the time to speak with the Social Impact Review today about the critical topic of technology, human rights, and racism. In your view, what are some of the reasons racism exists in technology today?
Cat Wade: There are so many manifestations of racism in technology that to answer this question it is best to start with a particular case in mind. This will help us to see what subset of this depressingly far-reaching phenomenon we’re dealing with. At the Harvard Advanced Leadership Initiative Deep Dive on “Human Rights and Inequality in the Digital Century” earlier this year, I discussed the case of discriminatory Google Image searches, in particular, what happens when we search for something like ‘unprofessional hairstyle.’ For many users, this search will yield a slew of African American woman with natural hair or braids. Compare this to a search for ‘professional hairstyles’ that yields predominantly white women with slicked back hair in ponytails or a bun.
When we compare these results, we feel that an injustice has occurred: African Americans are being represented as unprofessional. Kate Crawford calls this kind of burden a representational harm. Thus, the representational harm is disproportionately impacting one group and not another. This is disparate impact discrimination. And when the difference between those impacted groups is race, we have an instance of one kind of racism: racial discrimination. (Note that disparate impact discrimination, unlike disparate treatment, need not involve intent).
So, why is this happening? As I see it, there are two main causes: first is what is profitable for Google to show us, and second is the unfortunate reality of racism in the analog world. These causes co-produce each other. The first cause invites us to remember that Google is, first and foremost, a profit-seeking business. You’d be forgiven for thinking that this business relies on guessing at what a user wants when they search for things. If you ask yourself what you want from a Google search, you’ll probably say ‘the most relevant results.’ But it turns out that relevancy isn’t always what grabs our attention or what might cause us to click on a link or copy an image. Keeping us coming back and click-happy is what will make Google money in the long run and as such, despite appearances (Google’s business also relies on maintaining the image of an objective purveyor of informational resources), relevancy isn’t always what we get. To summarize, the main problem is that when we’re shown results that comport with what we expect to see, we’re apt to interpret these results as relevant. This is why echo chambers are so difficult to detect.
This brings us to the second cause: the reality of racism in the analog world. In many ways, search results hold up a mirror to the way the world is. If the world is racist, then search results will be racist too. Clicks make money, clicks track expectations, and expectations mirror the world. When the world is racist, racism is clickable and therefore profitable. The alternative would be to optimize search results not for clickability, or popularity, but for something else. This something else could be relevancy in a more specified sense: showing the actual range of possibilities for some search result. Or, we could be even bolder in vision and imagine optimizing for representing the world the way we would like it to be rather than the way it is – an aspirational approach. This might involve potentially overrepresenting traditionally underrepresented groups.
Lynch: How does racism manifest in technology and what are its effects?
Wade: Racism manifests at every stage of technology: from the databases that feed algorithmic technologies, to the hiring practices that make up the technological workforce, to the design processes for technological hardware. Many of these manifestations are systemic, meaning they are not perpetrated by individuals, but rather are enacted via structural forces like company policies or even cultural norms. Take the case highlighted by Ruha Benjamin of a hotel’s automatic hand soap dispenser that failed to work for darker skin tones — the sensor simply did not recognize the presence of a hand and therefore did not dispense soap. No one at the soap dispenser company set out to make a product that didn’t work for people with darker skin (I’m hoping) and yet somehow the sensor was approved and put into production before anyone realized this flaw. How could this happen? It is most likely due to the fact that employees at the company, as well as product testers, were in fact lighter-skinned. Coupled with the fact that no one at the company thought about what happens if the product is being used by people who didn’t look like them, this discriminatory feature went unnoticed until it was out in the world. This illustrates the manifestation of systemic racism in technology.
The effects of racism in technology are as far-reaching as these manifestations. My own research centers, in part, on the potential for technology to augment our self-making practices. In this domain, racial disparity in access to technology represents not just a modern-day inconvenience but a barrier to the full range of possible human experience.
Lynch: Can you share with us what interventions are possible to address some of these effects?
Wade: Interventions need to happen at every level of our technological lives. Just as racism and the impact of racism can manifest in hiring, design, databases, individual and institutional decisions and so on, so too must our selected interventions be multi-faceted. One interesting proposal I’m drawn to picks up on the ideas of our algorithms acting like ‘mirrors to our world’ as I discussed previously. This implies that the intervention should have our algorithms reflect the world we want, rather than the world as it is.
Algorithms reflecting the world with all its prejudice are the result of a number of factors, including Google’s business model and user behavior. Another huge contributor to this mirror phenomenon is what is included in the databases that the algorithms are trained on. Let’s take another example: predictive policing algorithms. Predictive policing algorithms are used nationwide by police forces to direct their resources (i.e., police officers/cars/etc.) to the locations where crime is most likely to occur. The idea is that the police force won’t be ‘wasted’ patrolling areas where crime isn’t very likely to occur, and, maybe, will be able to anticipate crime before it happens by being on the scene. The algorithm works by using a database full of police reports from previous years. It models the future world based on these past reports and predicts for headquarters where the crime spots tend to be on Wednesday afternoons in June on hot days. The police force is sent to those locations, any reports are filed and fed back into the database, and “efficiency” reigns supreme yet again.
There are two extremely pernicious things to note about this algorithm that pertain to its database. The first is that we know that policing in the U.S. is discriminatory. Therefore, we know that any database made up of actual police reports will also reflect this. Of course, it will be hard to keep this in mind once we have complied such a database, since individual lists of dates, times and locations don’t look very discriminatory. Nevertheless, the database on which the algorithm models the future bakes in past discriminatory policing behavior. It therefore runs the risk of sending police to areas where that discriminatory behavior can reoccur, such as in predominantly Black or Hispanic neighborhoods. The algorithm looks like it removed bias from the decision (after all, it’s not like some individual at headquarters is directing the police force around — it’s a computer!) but really it’s reflecting the way the world is and has been, which is far from perfect. The second way in which this reflection in the database is pernicious is that it directs a kind of action, police being sent to certain locations, that has the potential to reinforce the discriminatory bias that already exists in the database. Once the police are sent to some location, they are incentivized to make an arrest or a stop which justifies the algorithmic decision that sent them there. Therefore, they are more likely, than they might have otherwise been, to write people up for even minor infractions. Once these reports are entered into the database, the algorithm is even more likely to send police there in the future. Thus, this is a case in which the algorithm causes a feedback loop that makes it difficult for us to move away from the way the world currently is.
One solution I’m intrigued by is to take back control from the feedback loop and feed our algorithms data that reflect the way we want the world to be. In the policing case this might mean ‘sanitizing’ our datasets to root out the bias that we know or suspect occurs. Thus, we would intentionally edit the dataset to include fewer entries from locations that we know to be overpoliced for discriminatory reasons. Another example, going back to the Google Image case I mentioned earlier, we might change our Google datasets to have an equal number of images from a diverse group of women for every kind of potential professional hairstyle. This might mean intentionally removing some of the images of white women with slicked back hair in ponytails or a bun to create a balance. This way the algorithm models the world we might want — a world with less discrimination or representational harm.
Undoubtedly this proposal faces a number of hurdles. First and foremost, it risks losing predictive accuracy. In the policing case, there’s no way to know how much of the dataset is biased because of racist policing. There will be areas where more crime does in fact occur. This might be due to racist housing policies, say, but have nothing to do with policing. If we made it so the algorithm didn’t send more police that way, we would run the risk of not preventing crime where it needs preventing and end up with probable false negatives. We still wouldn’t want a world in which crime happens because of racist housing policies, but if we made the algorithm representative of a world where no discrimination of any kind occurred, we might fail to send the support that different communities need because they might need it because of discrimination in some other domain. Thus, in instances where algorithms are used to make decisions about the distribution of resources, we must be extra vigilant not to up with this result.
What about the arguably less high stakes Google Image case? Here it might seem more straightforward to have the repository of images reflect the diverse professional workforce we aspire to, rather than the somewhat homogenous workforce we in fact have. Again, this seems promising at first, but it begs the question: what is the diverse professional workforce ‘we’ want? Who would decide this? Is it racially diverse? Gender diverse? Is it diverse in an equally distributed way?
You might be wondering why I’m discussing this intervention if it is so seemingly fraught! While I think there are limitations to this approach, I don’t think this is a reason to give up on it. There may be suitable low stakes situations where we can very precisely identify the influence of some discriminatory practice and in those cases perhaps we should experiment with this intervention. Beyond this, however, this intervention is illustrative of one of the reasons we should be hopeful about combatting racism in technology: algorithms are much more malleable and consistent than human decision makers. While we might require annual anti-bias training of decision-makers, there’s no way to guarantee that this is permeating their decision making, or to know that the influence is consistent between sessions. Moreover, human decision makers are not very good at knowing why they made a decision. We’re prone to post-hoc confabulation. In contrast, we can see exactly how an algorithm comes to make some decision and we can manipulate it for fairer outcomes — unless the algorithm is opaque, but that’s another issue! Though there will surely be many things to figure out, the possibility of exploiting the concretized nature of algorithmic decision-making is an intervention I am hopeful about.
Lynch: Finally, can you tell us about your work regarding social media platforms and how social media users are susceptible to discrimination at the hands of the algorithms that curate the content they receive?
Wade: Social media platforms are, unfortunately, like the Google Image search case but on steroids. Content-curating algorithms on these platforms not only run the risk of showing you content that perpetuates representational harms, but they do so in a way that is intentionally aimed at capturing and holding your attention for as long as possible. Thus, the harmful effects discussed previously are amplified through repeated, ongoing exposure. In many cases this manipulation of the user’s attention is completely unnoticed. My work seeks to show how this restriction of our autonomy characterizes many of our experiences of the digital lifeworld. I argue that active participation in the construction of digital norms is one way to combat this. Norms such as ‘cancel culture’ that have sprung up ‘organically,’ or ‘passively,’ have been shown to have disastrous consequences. Instead of reverting to trial by fire in the (digital) public square we should all actively work to build norms, and ethics, that will make the digital lifeworld more just. We can’t just hope this will work out in the wash.
Lynch: Thank you very much Cat for your time and your insights today on this critical issue.
About the Author:
Susan Carney Lynch, Dr.PH, is a Harvard ALI Senior Fellow and Editor-in-Chief of the Social Impact Review. Prior to ALI, Susan spent 20 years at the United States Department of Justice as Senior Counsel for Elder Justice, where she led federal civil long-term care quality prosecutions nationwide and elder justice policy work. Susan has also had adjunct faculty appointments teaching health, law, and policy at law and public health schools for the past two decades.
This interview has been edited for length and clarity.