EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-SA 4.0

Goodbye human annotators? Content analysis of social policy debates using ChatGPT

Erwin Gielens, Jakub Sowula, Philip Leifeld

Journal of Social Policy · 2026-04 · 55 (2) · 385-404

DOI: 10.1017/s0047279424000382 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Social Policy, online erschienen am 03.01.2025. Verfasst von Erwin Gielens und 2 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 55 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-SA 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Content analysis is a valuable tool for analysing policy discourse, but annotation by humans is costly and time consuming. ChatGPT is a potentially valuable tool to partially automate content analysis for policy debates, largely replacing human annotators. We evaluate ChatGPT’s ability to classify documents using pre-defined argument descriptions, comparing its performance with human annotators for two policy debates: the Universal Basic Income debate on Dutch Twitter (2014–2016) and the pension reforms debate in German newspapers (1993–2001). We use the API (GPT-4 Turbo) and user interface version (GPT-4) and evaluate multiple performance metrics (accuracy, precision and recall). ChatGPT is highly reliable and accurate in classifying pre-defined arguments across datasets. However, precision and recall are much lower, and vary strongly between arguments. These results hold for both datasets, despite differences in language and media type. Moreover, the cut-off method proposed in this paper may aid researchers in navigating the trade-off between detection and noise. Overall, we do not (yet) recommend a blind application of ChatGPT to classify arguments in policy debates. Those interested in adopting this tool should manually validate bot classifications before using them in further analyses. At least for now, human annotators are here to stay.

Zitieren

APA

Gielens, E., Sowula, J., & Leifeld, P. (2026). Goodbye human annotators? Content analysis of social policy debates using ChatGPT. Journal of Social Policy, 55(2), 385-404. https://doi.org/10.1017/s0047279424000382

Vancouver

Gielens E, Sowula J, Leifeld P. Goodbye human annotators? Content analysis of social policy debates using ChatGPT. Journal of Social Policy. 2026;55(2):385-404. doi:10.1017/s0047279424000382

BibTeX
@article{Gielens2026,
  author = {Gielens, Erwin and Sowula, Jakub and Leifeld, Philip},
  title = {Goodbye human annotators? Content analysis of social policy debates using ChatGPT},
  journal = {Journal of Social Policy},
  year = {2026},
  volume = {55},
  number = {2},
  pages = {385--404},
  doi = {10.1017/s0047279424000382}
}
Kostenlos starten →