EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Large language models as a substitute for human experts in annotating political text

Michael Heseltine, Bernhard Clemm von Hohenberg

Research & Politics · 2024-01 · 11 (1)

DOI: 10.1177/20531680241236239 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Research & Politics, online erschienen am 28.02.2024. Verfasst von Michael Heseltine und 1 weiteren Autor:innen. Die Arbeit wurde bisher 105-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large-scale text analysis has grown rapidly as a method in political science and beyond. To date, text-as-data methods rely on large volumes of human-annotated training examples, which place a premium on researcher resources. However, advances in large language models (LLMs) may make automated annotation increasingly viable. This paper tests the performance of GPT-4 across a range of scenarios relevant for analysis of political text. We compare GPT-4 coding with human expert coding of tweets and news articles across four variables (whether text is political, its negativity, its sentiment, and its ideology) and across four countries (the United States, Chile, Germany, and Italy). GPT-4 coding is highly accurate, especially for shorter texts such as tweets, correctly classifying texts up to 95% of the time. Performance drops for longer news articles, and very slightly for non-English text. We introduce a ‘hybrid’ coding approach, in which disagreements of multiple GPT-4 runs are adjudicated by a human expert, which boosts accuracy. Finally, we explore downstream effects, finding that transformer models trained on hand-coded or GPT-4-coded data yield almost identical outcomes. Our results suggest that LLM-assisted coding is a viable and cost-efficient approach, although consideration should be given to task complexity.

Zitieren

APA

Heseltine, M., & Hohenberg, B. C. V. (2024). Large language models as a substitute for human experts in annotating political text. Research & Politics, 11(1). https://doi.org/10.1177/20531680241236239

Vancouver

Heseltine M, Hohenberg BCV. Large language models as a substitute for human experts in annotating political text. Research & Politics. 2024;11(1). doi:10.1177/20531680241236239

BibTeX
@article{Heseltine2024,
  author = {Heseltine, Michael and Hohenberg, Bernhard Clemm von},
  title = {Large language models as a substitute for human experts in annotating political text},
  journal = {Research & Politics},
  year = {2024},
  volume = {11},
  number = {1},
  doi = {10.1177/20531680241236239}
}
Kostenlos starten →