Assessing the Reliability of Large Language Models for Deductive Qualitative Coding: A Comparative Intervention Study with ChatGPT
Proceedings of the Association for Information Science and Technology · 2025-10 · 62 (1) · 275-285
DOI: 10.1002/pra2.1255 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the Association for Information Science and Technology, online erschienen am 16.10.2025. Verfasst von Angjelin Hila und 1 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 34 Literaturangaben.
Abstract (Originalsprache)
In this study we investigate the use of large language models (LLMs), specifically ChatGPT, for structured deductive qualitative coding. While most current research emphasizes inductive coding applications, we address the underexplored potential of LLMs to perform deductive classification tasks aligned with established human‐coded schemes. Using the Comparative Agendas Project (CAP) Master Codebook, we classified U.S. Supreme Court case summaries into 21 major policy domains. We tested four intervention methods: zero‐shot, few‐shot, definition‐based, and a novel Step‐by‐Step Task Decomposition strategy, across repeated samples. Performance was evaluated using standard classification metrics (accuracy, F1‐score, Cohen's κ , Krippendorff's α ), and construct validity was assessed using chi‐squared tests and Cramér's V. Chi‐squared and effect size analyses confirmed that intervention strategies significantly influenced classification behavior, with Cramér's V values ranging from 0.359 to 0.613, indicating moderate to strong shifts in classification patterns. The Step‐by‐Step Task Decomposition strategy achieved the strongest reliability (accuracy = 0.775, κ = 0.744, α = 0.746), achieving thresholds for substantial agreement. Despite the semantic ambiguity within case summaries, ChatGPT displayed stable agreement across samples, including high F1 scores in low‐support subclasses. These findings demonstrate that with targeted, custom‐tailored interventions LLMs can achieve reliability levels suitable for integration into rigorous qualitative coding workflows.
Zitieren
APA
Hila, A., & Hauser, E. (2025). Assessing the Reliability of Large Language Models for Deductive Qualitative Coding: A Comparative Intervention Study with ChatGPT. Proceedings of the Association for Information Science and Technology, 62(1), 275-285. https://doi.org/10.1002/pra2.1255
Vancouver
Hila A, Hauser E. Assessing the Reliability of Large Language Models for Deductive Qualitative Coding: A Comparative Intervention Study with ChatGPT. Proceedings of the Association for Information Science and Technology. 2025;62(1):275-285. doi:10.1002/pra2.1255
BibTeX
@article{Hila2025,
author = {Hila, Angjelin and Hauser, Elliott},
title = {Assessing the Reliability of Large Language Models for Deductive Qualitative Coding: A Comparative Intervention Study with ChatGPT},
journal = {Proceedings of the Association for Information Science and Technology},
year = {2025},
volume = {62},
number = {1},
pages = {275--285},
doi = {10.1002/pra2.1255}
}
EUVIMED