EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

ChatGPT outperforms crowd workers for text-annotation tasks

Fabrizio Gilardi iD, Meysam Alizadeh iD, Maël Kubli iD

Proceedings of the National Academy of Sciences · 25.07.2023 · 120 (30)

DOI: 10.1073/pnas.2305016120 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the National Academy of Sciences, online erschienen am 18.07.2023. Verfasst von Fabrizio Gilardi und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Political Science, University of Zurich, Zurich 8050, Switzerland. Die Arbeit wurde bisher 1131-mal zitiert (Crossref) und nennt 15 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: EC | European Research Council (ERC).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Many NLP applications require manual text annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using four samples of tweets and news articles ( n = 6,183), we show that ChatGPT outperforms crowd workers for several annotation tasks, including relevance, stance, topics, and frame detection. Across the four datasets, the zero-shot accuracy of ChatGPT exceeds that of crowd workers by about 25 percentage points on average, while ChatGPT’s intercoder agreement exceeds that of both crowd workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003—about thirty times cheaper than MTurk. These results demonstrate the potential of large language models to drastically increase the efficiency of text classification.

Schlagwörter

Stichwörter der Autor:innen

ChatGPThuman annotationslarge language modelstext as datatext classification

Autor:innen und Institutionen

  • Department of Political Science, University of Zurich, Zurich 8050, Switzerland.

ORCID-Profile: Fabrizio Gilardi, Meysam Alizadeh, Maël Kubli

Förderung

  • EC | European Research Council (ERC)

Zitieren

APA

Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30). https://doi.org/10.1073/pnas.2305016120

Vancouver

Gilardi F, Alizadeh M, Kubli M. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences. 2023;120(30). doi:10.1073/pnas.2305016120

BibTeX
@article{Gilardi2023,
  author = {Gilardi, Fabrizio and Alizadeh, Meysam and Kubli, Maël},
  title = {ChatGPT outperforms crowd workers for text-annotation tasks},
  journal = {Proceedings of the National Academy of Sciences},
  year = {2023},
  volume = {120},
  number = {30},
  doi = {10.1073/pnas.2305016120}
}
Kostenlos starten →