Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
Political Analysis · 2024-10 · 32 (4) · 401-416
DOI: 10.1017/pan.2024.5 ↗Kurz gefasst
Fachartikel in der Zeitschrift Political Analysis, online erschienen am 17.05.2024. Verfasst von James Bisbee und 4 weiteren Autor:innen. Die Arbeit wurde bisher 199-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large language models (LLMs) offer new research possibilities for social scientists, but their potential as “synthetic data” is still largely unknown. In this paper, we investigate how accurately the popular LLM ChatGPT can recover public opinion, prompting the LLM to adopt different “personas” and then provide feeling thermometer scores for 11 sociopolitical groups. The average scores generated by ChatGPT correspond closely to the averages in our baseline survey, the 2016–2020 American National Election Study (ANES). Nevertheless, sampling by ChatGPT is not reliable for statistical inference: there is less variation in responses than in the real surveys, and regression coefficients often differ significantly from equivalent estimates obtained using ANES data. We also document how the distribution of synthetic responses varies with minor changes in prompt wording, and we show how the same prompt yields significantly different results over a 3-month period. Altogether, our findings raise serious concerns about the quality, reliability, and reproducibility of synthetic survey data generated by LLMs.
Zitieren
APA
Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024). Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis, 32(4), 401-416. https://doi.org/10.1017/pan.2024.5
Vancouver
Bisbee J, Clinton JD, Dorff C, Kenkel B, Larson JM. Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis. 2024;32(4):401-416. doi:10.1017/pan.2024.5
BibTeX
@article{Bisbee2024,
author = {Bisbee, James and Clinton, Joshua D. and Dorff, Cassy and Kenkel, Brenton and Larson, Jennifer M.},
title = {Synthetic Replacements for Human Survey Data? The Perils of Large Language Models},
journal = {Political Analysis},
year = {2024},
volume = {32},
number = {4},
pages = {401--416},
doi = {10.1017/pan.2024.5}
}
EUVIMED