Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts
Political Analysis · 2026-04 · 34 (2) · 188-204
DOI: 10.1017/pan.2025.10017 ↗Kurz gefasst
Fachartikel in der Zeitschrift Political Analysis, online erschienen am 19.09.2025. Verfasst von Andrew Halterman und 1 weiteren Autor:innen. Die Arbeit wurde bisher 27-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Codebooks—documents that operationalize concepts and outline annotation procedures—are used almost universally by social scientists when coding political texts. To code these texts automatically, researchers are increasingly turning to generative large language models (LLMs). However, there is limited empirical evidence on whether “off-the-shelf” LLMs faithfully follow real-world codebook operationalizations and measure complex political constructs with sufficient accuracy. To address this, we gather and curate three real-world political science codebooks—covering protest events, political violence, and manifestos—along with their unstructured texts and human-coded labels. We also propose a five-stage framework for codebook-LLM measurement: Preparing a codebook for both humans and LLMs, testing LLMs’ basic capabilities on a codebook, evaluating zero-shot measurement accuracy (i.e., off-the-shelf performance), analyzing errors, and further (parameter-efficient) supervised training of LLMs. We provide an empirical demonstration of this framework using our three codebook datasets and several pre-trained 7–12 billion open-weight LLMs. We find current open-weight LLMs have limitations in following codebooks zero-shot, but that supervised instruction-tuning can substantially improve performance. Rather than suggesting the “best” LLM, our contribution lies in our codebook datasets, evaluation framework, and guidance for applied researchers who wish to implement their own codebook-LLM measurement projects.
Zitieren
APA
Halterman, A., & Keith, K. A. (2026). Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts. Political Analysis, 34(2), 188-204. https://doi.org/10.1017/pan.2025.10017
Vancouver
Halterman A, Keith KA. Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts. Political Analysis. 2026;34(2):188-204. doi:10.1017/pan.2025.10017
BibTeX
@article{Halterman2026,
author = {Halterman, Andrew and Keith, Katherine A.},
title = {Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts},
journal = {Political Analysis},
year = {2026},
volume = {34},
number = {2},
pages = {188--204},
doi = {10.1017/pan.2025.10017}
}
EUVIMED