EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Large Language Models and the Wisdom of Small Crowds

Sean Trott

Open Mind · 20.05.2024 · 8 · 723-738

DOI: 10.1162/opmi_a_00144 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Open Mind, online erschienen am 20.05.2024. Verfasst von Sean Trott. Beteiligte Einrichtung: Department of Cognitive Science, University of California, San Diego, San Diego, CA, USA. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent advances in Large Language Models (LLMs) have raised the question of replacing human subjects with LLM-generated data. While some believe that LLMs capture the “wisdom of the crowd”—due to their vast training data—empirical evidence for this hypothesis remains scarce. We present a novel methodological framework to test this: the “number needed to beat” (NNB), which measures how many humans are needed for a sample’s quality to rival the quality achieved by GPT-4, a state-of-the-art LLM. In a series of pre-registered experiments, we collect novel human data and demonstrate the utility of this method for four psycholinguistic datasets for English. We find that NNB > 1 for each dataset, but also that NNB varies across tasks (and in some cases is quite small, e.g., 2). We also introduce two “centaur” methods for combining LLM and human data, which outperform both stand-alone LLMs and human samples. Finally, we analyze the trade-offs in data cost and quality for each approach. While clear limitations remain, we suggest that this framework could guide decision-making about whether and how to integrate LLM-generated data into the research pipeline.

Schlagwörter

Stichwörter der Autor:innen

ChatGPTdatasetlarge language modelspsycholinguistic resourcewisdom of the crowd

Autor:innen und Institutionen

  • Department of Cognitive Science, University of California, San Diego, San Diego, CA, USA.

Zitieren

APA

Trott, S. (2024). Large Language Models and the Wisdom of Small Crowds. Open Mind, 8, 723-738. https://doi.org/10.1162/opmi_a_00144

Vancouver

Trott S. Large Language Models and the Wisdom of Small Crowds. Open Mind. 2024;8:723-738. doi:10.1162/opmi_a_00144

BibTeX
@article{Trott2024,
  author = {Trott, Sean},
  title = {Large Language Models and the Wisdom of Small Crowds},
  journal = {Open Mind},
  year = {2024},
  volume = {8},
  pages = {723--738},
  doi = {10.1162/opmi_a_00144}
}
Kostenlos starten →