EUVIMEDEUVIMED entdecken
Crossref

The Mixed Subjects Design: Treating Large Language Models as Potentially Informative Observations

David Broska, Michael Howes, Austin van Loon

Sociological Methods & Research · 2025-08 · 54 (3) · 1074-1109

DOI: 10.1177/00491241251326865 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sociological Methods & Research, online erschienen am 22.04.2025. Verfasst von David Broska und 2 weiteren Autor:innen. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 61 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) provide cost-effective but possibly inaccurate predictions of human behavior. Despite growing evidence that predicted and observed behavior are often not interchangeable , there is limited guidance on using LLMs to obtain valid estimates of causal effects and other parameters. We argue that LLM predictions should be treated as potentially informative observations, while human subjects serve as a gold standard in a mixed subjects design . This paradigm preserves validity and offers more precise estimates at a lower cost than experiments relying exclusively on human subjects. We demonstrate—and extend—prediction-powered inference (PPI), a method that combines predictions and observations. We define the PPI correlation as a measure of interchangeability and derive the effective sample size for PPI. We also introduce a power analysis to optimally choose between informative but costly human subjects and less informative but cheap predictions of human behavior. Mixed subjects designs could enhance scientific productivity and reduce inequality in access to costly evidence.

Zitieren

APA

Broska, D., Howes, M., & Loon, A. V. (2025). The Mixed Subjects Design: Treating Large Language Models as Potentially Informative Observations. Sociological Methods & Research, 54(3), 1074-1109. https://doi.org/10.1177/00491241251326865

Vancouver

Broska D, Howes M, Loon AV. The Mixed Subjects Design: Treating Large Language Models as Potentially Informative Observations. Sociological Methods & Research. 2025;54(3):1074-1109. doi:10.1177/00491241251326865

BibTeX
@article{Broska2025,
  author = {Broska, David and Howes, Michael and Loon, Austin van},
  title = {The Mixed Subjects Design: Treating Large Language Models as Potentially Informative Observations},
  journal = {Sociological Methods & Research},
  year = {2025},
  volume = {54},
  number = {3},
  pages = {1074--1109},
  doi = {10.1177/00491241251326865}
}
Kostenlos starten →