Understanding Gender and Racial Bias in AI
A Conversation with Dr. Alex Hanna
Dr. Alex Hanna is Director of Research at the Distributed AI Research Institute. A sociologist by training, her work centers on the data used in new computational technologies, and the ways in which these data exacerbate racial, gender, and class inequality. She also works in the area of social movements, focusing on the dynamics of anti-racist campus protests in the U.S. and Canada. Dr. Hanna has published widely in top-tier venues across the social sciences, including the journals Mobilization, American Behavioral Scientist, and Big Data & Society, and top-tier computer science conferences such as CSCW, FAccT, and NeurIPS. Dr. Hanna serves as co-chair of Sociologists for Trans Justice and as a Senior Fellow at the Center for Applied Transgender Studies. FastCompany included Dr. Hanna as part of their 2021 Queer 50, and she has been featured in the Cal Academy of Sciences New Science exhibit, which highlights queer and trans scientists of color.
Gina Lázaro: Please tell us about DAIR. What is the organization’s mission and what type of research and work does DAIR do?
Alex Hanna: DAIR is the Distributed AI Research Institute. It was founded by Dr. Timnit Gebru in 2021. The idea behind DAIR is we wanted to have a space, an institute, to do research on AI technology without the constraints of being either within a large tech corporation or within an academic lab that is often funded by large tech money. We wanted to start from the premise, which is unusual for an AI institute, that the use of AI is not inevitable, harms from AI are preventable, and AI can be beneficial, but we recognize that a lot of the purposes AI is being used for currently are for commercial, military, and surveillance. So, we have four principles. First, we focus on community and non-exploitation knowing that the communities that are disproportionately affected by AI are the ones that are not in the room, and if we want to include the voices of the people, we should do our work in a way that is not exploitative. We do not want to say “Oh, we talked to just a few people, and we didn't compensate them for their work.” Second, we have comprehensive processes to ensure that we conduct our research in a principled, deliberate way that is not ad hoc. We need to be accountable to the communities as well. Third, we take a pragmatic approach to research focusing on making AI in sociotechnical systems beneficial, but also knowing that AI is not the answer much of the time. Our work is focused on understanding what AI could be used for, but also putting priority on the desires of the communities that we serve. Fourth, for our own research, we don't want to be killing ourselves being tied to a quarterly publish or perish model. We know we need to take things slowly since, with our emphasis on community, it's a process of building trust and relationships, and being intentional towards our goals.
Lázaro: You mentioned the need for independent AI research that is not funded by tech companies. Could you speak a bit more to the importance of independence?
Hanna: Absolutely. Right now, AI research is mostly in the domains of large tech companies, elite universities, or specialty labs like OpenAI or Anthropic which have these really big VC term sheets and are doing things which they consider to be either general purpose AI, Artificial General Intelligence (AGI) or something of that nature. But this way of working is not allowing AI researchers to be independent or to be community focused. Instead, AI research basically is supporting bad markets. For example, when it comes to corporate uses, most of Google and Facebook’s uses for development of AI are either supporting their core businesses or towards military or security purposes. Also, there are institutions like Clearview AI, which is oriented towards facial recognition on a massive scale for law enforcement, or ShotSpotter, which detects things like gunshots, or purports to, but is mostly just used to reinforce policing of communities of color. If we want independent research, we must think about what it means to do AI research that starts from different premises, research from the position of what community members say they need, and what is affecting those communities.
Lázaro: Thanks for that clarification. According to a Pew Research study in March 2022, 37% of Americans are “more concerned than excited” and 45% are “equally concerned and excited” about the increased use of AI in daily life. Among the concerns are gender and racial bias in AI. What are the reasons causing bias in AI systems? To better understand the issues, could you give us some examples of when using AI systems has resulted in gender and racial bias?
Hanna: Sure, gender and racial biases are the function of a few different things in AI systems. If you talk to computer scientists, they will say the problem is with the data, which is a reductive answer because it does not provide the full story. AI bias also occurs because of who is in the room formulating and framing the problem. For instance, if you start from the premise that facial recognition is biased against Black women, which Dr. Gebru and Dr. Joy Buolamwini showed in their Gender Shades study is correct, one of the practical solutions would be to increase the number of Black women in the data set. However, the other larger concern is that facial recognition itself is a racist technology, because it is disproportionately leveraged in communities of color, in the Global South and at the borders. So, even the construction of that problem shows the potential of a racist outcome.
A second understanding is that AI has adverse outcomes because the formulation of the variables used are not attentive to larger institutional structures. For instance, there is a study done by Ziad Obermeyer that showed an AI algorithm used to disperse medical funding has biases against Black patients because one of the input variables was the amount of expenditure individuals previously spent on health care versus actual medical data. This is a problem because Black individuals spend less on health care in the early stages of illness, so the AI algorithm determined that Black patients needed less health funding going forward based on flawed and incomplete information. The basic issue is that the problem and variables were framed incorrectly, and the algorithm didn’t know how spending was dispersed nor what kind of institutions surrounded the issue.
Now, there is also the problem with data because data itself is how machine learning models are trained. If you are just gathering data from the internet without attention to embedded gender and race bias in the U.S., that is a significant concern. In other locations around the world, we also see bias in language, religion, and caste. We usually talk about gender and race bias and their intersections in the U.S. context, multiple causes of bias and how bias causes harm.
Lázaro: Is there any way to ensure that the underlying data that is used in a data set is accurate and non-biased?
Hanna: There are methods of trying to assess how much bias exists within an algorithm such as computational methods that depend on whether the modality is images or text. For text, you can compare the co-occurrences and differences of racialized or gendered words and identify associations and co-occurrences with negative words. For instance, finding how many times “man” or “woman” co-occurs with a professionalized term like “doctor” or “nurse.” But I would say that there are no surefire ways of debiasing a data set because bias is always going to be embedded in some way. If you try to “debias” a language dataset for a gendered dimension or a racial dimension, for instance, you may also miss the intersection of these, or you miss debiasing by ableist language. At some point, you also must state your values up front, state how you are choosing to encode individuals, and explicitly talk about how you are defining parity. Some things cannot be expressed merely in terms of bias. The way bias is often discussed is as an undue preference for one class versus another, but that requires one to focus on defined classes. For instance, in gender bias, the data set standard is usually only framed in terms of male versus female, and that already erases nonbinary people and people who do not fit within a Western gender binary. In terms of racial bias, under U.S. discrimination law, racial bias is meant to be symmetric. We tend to think about ensuring that there is a definition of fairness that purports to need a requisite number of Black individuals in the population compared to a general population. For instance, the way that the U.S. has successfully argued — let's say for affirmative action — has been with the inclusion of men, because it is symmetric. But if you are saying, we need a way to ameliorate past harms, such as the U.S. legacy of chattel slavery or the decades of carceral immigration schemes, that also encodes a notion of what bias is and how you counter that bias. I say all the time, there is no such thing as a bias-free data set. You can mitigate bias by making value commitments expressive and outlining them from the outset. We need to start there.
Lázaro: Can AI systems be used to improve gender and racial equality for better outcomes? For example, is there a way for AI to be a positive force for good, and what would it take to improve the AI systems to become fair and equitable?
Hanna: We think a lot about AI for more beneficial purposes at DAIR. A project we are working on is focusing on using AI tools for the mitigation of harms that are existing on current social media platforms. For instance, we have been developing a project on identifying hate speech and disinformation against the Tigrayan minority in Ethiopia, an outcome of the Ethiopian civil war. It is a failure of tech companies, especially platforms, to do the sufficient work in mitigating the abuses on the platforms targeting activists, advocates, journalists, and lawyers. Companies are having a hard time doing content moderation at scale since they have not sufficiently provided machine translation tools to translate from Amharic and Tigrinya to English nor have individuals who can read those texts in the original language.
One thing that could be developed, and we are helping to develop, is a set of natural language processing tools that could assess and try to detect, at least at first blush, the elements of misinformation, disinformation, and hate speech. This work is already complicated in English, as there is no good agreement on what constitutes hate speech in AI research, so the complexity increases in other languages. Another way is to use some of the tools that we know can detect bias to assess data sets. That is not even necessarily AI, but it is a way of assessing how unbalanced the data set may be. It might be a place to start but it is certainly not the end.
Lázaro: What roles and responsibilities do businesses, data scientists, engineers, and society overall have to ensure that development and implementation of AI systems is done responsibly?
Hanna: Businesses that want to be socially beneficial need to do their due diligence in understanding the impact of deploying their systems. There are models that have been suggested in the space, like human rights impact assessments, and understanding and ensuring, with the release of a product or something into the public domain, that the system will not harm a marginalized group or population. Certainly, business has the responsibility to do that, and their organizations must be structured such that people who are doing this work are protected. For instance, one of the reasons why the Ethical AI team at Google exploded one and a half years ago is because Dr. Gebru, the founder of DAIR, and Dr. Meg Mitchell, the founder of the Ethical AI team, were forced out by Google for doing the work to detect harm in a class of algorithms called language models and were not protected for it. Businesses need to be committed to act responsibly, especially when it may affect their bottom line, and data scientists and engineers must be protected for doing the work. Moreover, business also needs to have people who understand social context on staff, because it is not just about the technology. This work is about the interaction of technology and society. I always argue this as I’m a sociologist who has studied technology for over 15 years. This interaction is going to be complicated, and it is going to take a different knowledge far beyond the technical, as the experts need to understand the legacies of things like institutional racism, sexism, homophobia, transphobia, ableism, U.S. centrism, and colonialism. It’s understanding where that rubber meets the road and how to fit within doing business responsibly that is incredibly important.
Lázaro: How can businesses that have the desire to do better gain the knowledge that you just laid out?
Hanna: That's the hard question as the space is still developing. I would say that if businesses are oriented towards building those skills in-house, then they need to hire expansively. They need to find people who have been thinking at this intersection of technology and society, and really think about what it would mean to build ethical tooling and ethical teams, and then allow those teams to have actual power to influence the direction of product. I have not seen too many organizations do that well. Twitter might have been one of the better ones as their ethics team (called META for Machine Learning, Ethics, Transparency and Accountability) is led by a social scientist, Rumman Chowdhury, who has significant experience in technology. It will be interesting to see what path they go on now with their pending purchase by Elon Musk.
There are organizations that are emerging that are focused on doing consulting around ethical practices. For instance, Erika Cheung, who was the Theranos whistleblower, has started her own institution called the Ethics in Entrepreneurship. There are models that can be replicated or improved upon for corporate social responsibility or potentially around human rights impact assessment, but I don't think that it is sufficient to do it in-house. If companies want to do right by it, in-house is a place to start but, ultimately, there must be governance and regulation. Companies cannot effectively self-regulate; there needs to be regulatory oversight.
Lázaro: What should regulation look like, and what is the status of regulation in the U.S. and globally?
Hanna: The U.S. and the E.U. are the two dominant regions that are in these discussions. The E.U. is further along between the General Data Protection Regulation (GDPR) and the AI Act. In the U.S., the Algorithmic Accountability Act was proposed in 2019 and has been improved upon in the latest session by Senators Booker and Wyden and Representative Clark. This bill ensures that organizations have all their models undergo review and audit for more transparency. Also, it provides the FTC, as the regulator, with sufficient enforcement capabilities to assess the outcomes of the models. This would be akin to being able to inspect the labs or outputs done on a medical level, so it is a promising piece of legislation. But I also think we need national privacy and data governance laws. These do not exist in the U.S. nationally; rather we have a patchwork between California, Virginia, with the strongest one in Illinois.
We should also regulate the use of data, whether that is private data being used by companies, or data that is the representation of people, like their images or their utterances online. There is still much grey space that current U.S. legislation or regulation has not sufficiently addressed. In this rapidly advancing space, oversight must keep up with the pace of innovation.
Lázaro: Alex, that is a great overview on the issues with AI. Thank you so much for your time and expertise.
About the Author:
Gina Lázaro is a Harvard Advanced Leadership Initiative Senior Fellow. She has a background in global consumer products marketing with her last role as Chief Marketing Officer at FGX International, a subsidiary of Essilor. Gina serves on the advisory council of HighSight, a non-profit focused on educational opportunities for low-income African American and Latino youth, as well as on the board of The Canales Project, a non-profit arts and advocacy organization.
This interview has been edited for length and clarity.