EUVIMEDEUVIMED entdecken
Crossref

Assessing the Reliability of Large Language Models for Deductive Qualitative Coding: A Comparative Intervention Study with ChatGPT

Angjelin Hila, Elliott Hauser

Proceedings of the Association for Information Science and Technology · 2025-10 · 62 (1) · 275-285

DOI: 10.1002/pra2.1255 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the Association for Information Science and Technology, online erschienen am 16.10.2025. Verfasst von Angjelin Hila und 1 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 34 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this study we investigate the use of large language models (LLMs), specifically ChatGPT, for structured deductive qualitative coding. While most current research emphasizes inductive coding applications, we address the underexplored potential of LLMs to perform deductive classification tasks aligned with established human‐coded schemes. Using the Comparative Agendas Project (CAP) Master Codebook, we classified U.S. Supreme Court case summaries into 21 major policy domains. We tested four intervention methods: zero‐shot, few‐shot, definition‐based, and a novel Step‐by‐Step Task Decomposition strategy, across repeated samples. Performance was evaluated using standard classification metrics (accuracy, F1‐score, Cohen's κ , Krippendorff's α ), and construct validity was assessed using chi‐squared tests and Cramér's V. Chi‐squared and effect size analyses confirmed that intervention strategies significantly influenced classification behavior, with Cramér's V values ranging from 0.359 to 0.613, indicating moderate to strong shifts in classification patterns. The Step‐by‐Step Task Decomposition strategy achieved the strongest reliability (accuracy = 0.775, κ = 0.744, α = 0.746), achieving thresholds for substantial agreement. Despite the semantic ambiguity within case summaries, ChatGPT displayed stable agreement across samples, including high F1 scores in low‐support subclasses. These findings demonstrate that with targeted, custom‐tailored interventions LLMs can achieve reliability levels suitable for integration into rigorous qualitative coding workflows.

Zitieren

APA

Hila, A., & Hauser, E. (2025). Assessing the Reliability of Large Language Models for Deductive Qualitative Coding: A Comparative Intervention Study with ChatGPT. Proceedings of the Association for Information Science and Technology, 62(1), 275-285. https://doi.org/10.1002/pra2.1255

Vancouver

Hila A, Hauser E. Assessing the Reliability of Large Language Models for Deductive Qualitative Coding: A Comparative Intervention Study with ChatGPT. Proceedings of the Association for Information Science and Technology. 2025;62(1):275-285. doi:10.1002/pra2.1255

BibTeX
@article{Hila2025,
  author = {Hila, Angjelin and Hauser, Elliott},
  title = {Assessing the Reliability of Large Language Models for Deductive Qualitative Coding: A Comparative Intervention Study with ChatGPT},
  journal = {Proceedings of the Association for Information Science and Technology},
  year = {2025},
  volume = {62},
  number = {1},
  pages = {275--285},
  doi = {10.1002/pra2.1255}
}
Kostenlos starten →