Auditing large language models for race & gender disparities: Implications for artificial intelligence-based hiring
Behavioral Science & Policy · 2024-10 · 10 (2) · 46-55
DOI: 10.1177/23794607251320229 ↗Kurz gefasst
Fachartikel in der Zeitschrift Behavioral Science & Policy, online erschienen am 11.03.2025. Verfasst von Johann D. Gaebler und 3 weiteren Autor:innen. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 63 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Rapid advances in artificial intelligence (AI), including large language models (LLMs) with abilities that rival those of human experts on a wide array of tasks, are reshaping how people make important decisions. At the same time, critics worry that LLMs may inadvertently discriminate against some groups. To address these concerns, recent regulations call for auditing the LLMs used in important decisions such as hiring. But neither current regulations nor the scientific literature offers clear guidance on how to conduct these audits. In this article, we propose and investigate one approach for auditing algorithms: correspondence experiments , a widely applied tool for detecting bias in human judgments. We applied this method to a range of LLMs instructed to rate job candidates using a novel data set of job applications for K-12 teaching positions in a large American public school district. By altering the application materials to imply that candidates are members of specific demographic groups, we measured the extent to which race and gender influenced the LLMs’ ratings of the candidates’ suitability. We found moderate race and gender disparities, with the models slightly favoring women and non-White candidates. This pattern persisted across several variations in our experiment. It is unclear what might be driving these disparities, but we hypothesize that they stem from posttraining efforts, which are part of the LLM training process and intended to correct biases in these models. We conclude by discussing the limitations of correspondence experiments for auditing algorithms.
Zitieren
APA
Gaebler, J. D., Goel, S., Huq, A., & Tambe, P. (2024). Auditing large language models for race & gender disparities: Implications for artificial intelligence-based hiring. Behavioral Science & Policy, 10(2), 46-55. https://doi.org/10.1177/23794607251320229
Vancouver
Gaebler JD, Goel S, Huq A, Tambe P. Auditing large language models for race & gender disparities: Implications for artificial intelligence-based hiring. Behavioral Science & Policy. 2024;10(2):46-55. doi:10.1177/23794607251320229
BibTeX
@article{Gaebler2024,
author = {Gaebler, Johann D. and Goel, Sharad and Huq, Aziz and Tambe, Prasanna},
title = {Auditing large language models for race & gender disparities: Implications for artificial intelligence-based hiring},
journal = {Behavioral Science & Policy},
year = {2024},
volume = {10},
number = {2},
pages = {46--55},
doi = {10.1177/23794607251320229}
}
EUVIMED