EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Large language models display human-like social desirability biases in Big Five personality surveys

Aadesh Salecha iD, Molly E Ireland iD, Shashanka Subrahmanya, João Sedoc, Lyle H Ungar iD, Johannes C Eichstaedt iD

PNAS Nexus · 28.11.2024 · 3 (12)

DOI: 10.1093/pnasnexus/pgae533 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PNAS Nexus, online erschienen am 17.12.2024. Verfasst von Aadesh Salecha und 5 weiteren Autor:innen. Beteiligte Einrichtung: Institute for Human-Centered AI, Stanford University, Palo Alto, CA 94305, USA. Die Arbeit wurde bisher 56-mal zitiert (Crossref) und nennt 15 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Stanford Institute for Human-Centered AI, Artificial Intelligence in Medicine, Imaging at Stanford University.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) are becoming more widely used to simulate human participants and so understanding their biases is important. We developed an experimental framework using Big Five personality surveys and uncovered a previously undetected social desirability bias in a wide range of LLMs. By systematically varying the number of questions LLMs were exposed to, we demonstrate their ability to infer when they are being evaluated. When personality evaluation is inferred, LLMs skew their scores towards the desirable ends of trait dimensions (i.e. increased extraversion, decreased neuroticism, etc.). This bias exists in all tested models, including GPT-4/3.5, Claude 3, Llama 3, and PaLM-2. Bias levels appear to increase in more recent models, with GPT-4’s survey responses changing by 1.20 (human) SD and Llama 3’s by 0.98 SD, which are very large effects. This bias remains after question order randomization and paraphrasing. Reverse coding the questions decreases bias levels but does not eliminate them, suggesting that this effect cannot be attributed to acquiescence bias. Our findings reveal an emergent social desirability bias and suggest constraints on profiling LLMs with psychometric tests and on this use of LLMs as proxies for human participants.

Schlagwörter

Stichwörter der Autor:innen

AIcognitive biasemergent behaviorlarge language modelspsychometrics

Autor:innen und Institutionen

  • Institute for Human-Centered AI, Stanford University, Palo Alto, CA 94305, USA.
  • Receptiviti, Toronto, ON M5G 2K8, Canada.
  • Leonard N. Stern School of Business, New York University, New York, NY 10012, USA.
  • Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104, USA.

ORCID-Profile: Aadesh Salecha, Molly E Ireland, Lyle H Ungar, Johannes C Eichstaedt

Förderung

  • Stanford Institute for Human-Centered AI
  • Artificial Intelligence in Medicine
  • Imaging at Stanford University

Zitieren

APA

Salecha, A., Ireland, M. E., Subrahmanya, S., Sedoc, J., Ungar, L. H., & Eichstaedt, J. C. (2024). Large language models display human-like social desirability biases in Big Five personality surveys. PNAS Nexus, 3(12). https://doi.org/10.1093/pnasnexus/pgae533

Vancouver

Salecha A, Ireland ME, Subrahmanya S, Sedoc J, Ungar LH, Eichstaedt JC. Large language models display human-like social desirability biases in Big Five personality surveys. PNAS Nexus. 2024;3(12). doi:10.1093/pnasnexus/pgae533

BibTeX
@article{Salecha2024,
  author = {Salecha, Aadesh and Ireland, Molly E and Subrahmanya, Shashanka and Sedoc, João and Ungar, Lyle H and Eichstaedt, Johannes C},
  title = {Large language models display human-like social desirability biases in Big Five personality surveys},
  journal = {PNAS Nexus},
  year = {2024},
  volume = {3},
  number = {12},
  doi = {10.1093/pnasnexus/pgae533}
}
Kostenlos starten →