EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Qualitative Coding with GPT-4

Xiner Liu, Andres Felipe Zambrano, Ryan S. Baker, Amanda Barany, Jaclyn Ocumpaugh, Jiayi Zhang, Maciej Pankiewicz, Nidhi Nasiar, Zhanlan Wei

Journal of Learning Analytics · 17.03.2025 · 12 (1) · 169-185

DOI: 10.18608/jla.2025.8575 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Learning Analytics, online erschienen am 17.03.2025. Verfasst von Xiner Liu und 8 weiteren Autor:innen. Die Arbeit wurde bisher 59-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This study explores the potential of the large language model GPT-4 as an automated tool for qualitative data analysis by educational researchers, exploring which techniques are most successful for different types of constructs. Specifically, we assess three different prompt engineering strategies — Zero-shot, Few-shot, and Few-shot with contextual information — as well as the use of embeddings. We do so in the context of qualitatively coding three distinct educational datasets: Algebra I semi-personalized tutoring session transcripts, student observations in a game-based learning environment, and debugging behaviours in an introductory programming course. We evaluated the performance of each approach based on its inter-rater agreement with human coders and explored how different methods vary in effectiveness depending on a construct’s degree of clarity, concreteness, objectivity, granularity, and specificity. Our findings suggest that while GPT-4 can code a broad range of constructs, no single method consistently outperforms the others, and the selection of a particular method should be tailored to the specific properties of the construct and context being analyzed. We also found that GPT-4 has the most difficulty with the same constructs than human coders find more difficult to reach inter-rater reliability on.

Zitieren

APA

Liu, X., Zambrano, A. F., Baker, R. S., Barany, A., Ocumpaugh, J., Zhang, J., Pankiewicz, M., Nasiar, N., & Wei, Z. (2025). Qualitative Coding with GPT-4. Journal of Learning Analytics, 12(1), 169-185. https://doi.org/10.18608/jla.2025.8575

Vancouver

Liu X, Zambrano AF, Baker RS, Barany A, Ocumpaugh J, Zhang J, et al. Qualitative Coding with GPT-4. Journal of Learning Analytics. 2025;12(1):169-185. doi:10.18608/jla.2025.8575

BibTeX
@article{Liu2025,
  author = {Liu, Xiner and Zambrano, Andres Felipe and Baker, Ryan S. and Barany, Amanda and Ocumpaugh, Jaclyn and Zhang, Jiayi and Pankiewicz, Maciej and Nasiar, Nidhi and Wei, Zhanlan},
  title = {Qualitative Coding with GPT-4},
  journal = {Journal of Learning Analytics},
  year = {2025},
  volume = {12},
  number = {1},
  pages = {169--185},
  doi = {10.18608/jla.2025.8575}
}
Kostenlos starten →