Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy
Science Advances · 08.11.2024 · 10 (45)
DOI: 10.1126/sciadv.adp1528 ↗Kurz gefasst
Fachartikel in der Zeitschrift Science Advances, erschienen am 08.11.2024. Verfasst von Philipp Schoenegger und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Management, London School of Economics and Political Science, Houghton Street, London WC2A 2AE, UK. Hauptthemen laut MEDLINE: Language. Die Arbeit wurde bisher 61-mal zitiert (Crossref) und nennt 67 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Human forecasting accuracy improves through the “wisdom of the crowd” effect, in which aggregated predictions tend to outperform individual ones. Past research suggests that individual large language models (LLMs) tend to underperform compared to human crowd aggregates. We simulate a wisdom of the crowd effect with LLMs. Specifically, we use an ensemble of 12 LLMs to make probabilistic predictions about 31 binary questions, comparing them with those made by 925 human forecasters in a 3-month tournament. We show that the LLM crowd outperforms a no-information benchmark and is statistically indistinguishable from the human crowd. We also observe human-like biases, such as the acquiescence bias. In another study, we find that LLM predictions (of GPT-4 and Claude 2) improve when exposed to the median human prediction, increasing accuracy by 17 to 28%. However, simply averaging human and machine forecasts yields more accurate results. Our findings suggest that LLM predictions can rival the human crowd’s forecasting accuracy through simple aggregation.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Management, London School of Economics and Political Science, Houghton Street, London WC2A 2AE, UK.
- Independent researcher, London, UK.
- Department of Physics, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA 02139, USA.
- Department of Psychology, Universidade São Francisco, Av. São Francisco de Assis, 218 - Jardim Sao Jose, Bragança Paulista-SP 12916-900, Brazil.
- Department of Psychology, University of Pennsylvania, 3720 Walnut St, Philadelphia, PA 19104, USA.
- The Wharton School, University of Pennsylvania, 3733 Spruce St, Philadelphia, PA 19104, USA.
ORCID-Profile: Philipp Schoenegger, Peter S Park, Philip E Tetlock
Zitieren
APA
Schoenegger, P., Tuminauskaite, I., Park, P. S., Bastos, R. V. S., & Tetlock, P. E. (2024). Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy. Science Advances, 10(45). https://doi.org/10.1126/sciadv.adp1528
Vancouver
Schoenegger P, Tuminauskaite I, Park PS, Bastos RVS, Tetlock PE. Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy. Science Advances. 2024;10(45). doi:10.1126/sciadv.adp1528
BibTeX
@article{Schoenegger2024,
author = {Schoenegger, Philipp and Tuminauskaite, Indre and Park, Peter S and Bastos, Rafael Valdece Sousa and Tetlock, Philip E},
title = {Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy},
journal = {Science Advances},
year = {2024},
volume = {10},
number = {45},
doi = {10.1126/sciadv.adp1528}
}
EUVIMED