Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research
Social Science Computer Review · 2026-06 · 44 (3) · 403-427
DOI: 10.1177/08944393251366243 ↗Kurz gefasst
Fachartikel in der Zeitschrift Social Science Computer Review, online erschienen am 08.08.2025. Verfasst von Hao Lin und 1 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 61 Literaturangaben.
Abstract (Originalsprache)
Large language models (LLMs) have the potential to revolutionize computational social science, particularly in automated textual analysis. In this paper, we conduct a systematic evaluation of the promises and risks associated with using LLMs for text classification tasks, using social movement studies as an example. We propose a framework for social scientists to incorporate LLMs into text annotation, either as the primary coding decision-maker or as a coding assistant. This framework offers researchers tools to develop the potential best-performing prompt, and to systematically examine and report the validity and reliability of LLMs as a methodological tool. Additionally, we evaluate and discuss its epistemic risks associated with validity, reliability, replicability, and transparency. We conclude with several practical guidelines for using LLMs in text annotation tasks and offer recommendations for more effectively communicating epistemic risks in research.
Zitieren
APA
Lin, H., & Zhang, Y. (2026). Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research. Social Science Computer Review, 44(3), 403-427. https://doi.org/10.1177/08944393251366243
Vancouver
Lin H, Zhang Y. Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research. Social Science Computer Review. 2026;44(3):403-427. doi:10.1177/08944393251366243
BibTeX
@article{Lin2026,
author = {Lin, Hao and Zhang, Yongjun},
title = {Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research},
journal = {Social Science Computer Review},
year = {2026},
volume = {44},
number = {3},
pages = {403--427},
doi = {10.1177/08944393251366243}
}
EUVIMED