Qualitative Coding with GPT-4
Journal of Learning Analytics · 17.03.2025 · 12 (1) · 169-185
DOI: 10.18608/jla.2025.8575 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Learning Analytics, online erschienen am 17.03.2025. Verfasst von Xiner Liu und 8 weiteren Autor:innen. Die Arbeit wurde bisher 59-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
This study explores the potential of the large language model GPT-4 as an automated tool for qualitative data analysis by educational researchers, exploring which techniques are most successful for different types of constructs. Specifically, we assess three different prompt engineering strategies — Zero-shot, Few-shot, and Few-shot with contextual information — as well as the use of embeddings. We do so in the context of qualitatively coding three distinct educational datasets: Algebra I semi-personalized tutoring session transcripts, student observations in a game-based learning environment, and debugging behaviours in an introductory programming course. We evaluated the performance of each approach based on its inter-rater agreement with human coders and explored how different methods vary in effectiveness depending on a construct’s degree of clarity, concreteness, objectivity, granularity, and specificity. Our findings suggest that while GPT-4 can code a broad range of constructs, no single method consistently outperforms the others, and the selection of a particular method should be tailored to the specific properties of the construct and context being analyzed. We also found that GPT-4 has the most difficulty with the same constructs than human coders find more difficult to reach inter-rater reliability on.
Zitieren
APA
Liu, X., Zambrano, A. F., Baker, R. S., Barany, A., Ocumpaugh, J., Zhang, J., Pankiewicz, M., Nasiar, N., & Wei, Z. (2025). Qualitative Coding with GPT-4. Journal of Learning Analytics, 12(1), 169-185. https://doi.org/10.18608/jla.2025.8575
Vancouver
Liu X, Zambrano AF, Baker RS, Barany A, Ocumpaugh J, Zhang J, et al. Qualitative Coding with GPT-4. Journal of Learning Analytics. 2025;12(1):169-185. doi:10.18608/jla.2025.8575
BibTeX
@article{Liu2025,
author = {Liu, Xiner and Zambrano, Andres Felipe and Baker, Ryan S. and Barany, Amanda and Ocumpaugh, Jaclyn and Zhang, Jiayi and Pankiewicz, Maciej and Nasiar, Nidhi and Wei, Zhanlan},
title = {Qualitative Coding with GPT-4},
journal = {Journal of Learning Analytics},
year = {2025},
volume = {12},
number = {1},
pages = {169--185},
doi = {10.18608/jla.2025.8575}
}
EUVIMED