Automated Coding of Communications in Collaborative Problem‐Solving Tasks Using ChatGPT
Journal of Educational Measurement · 2025-12 · 62 (4) · 809-837
DOI: 10.1111/jedm.70014 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Educational Measurement, online erschienen am 23.10.2025. Verfasst von Jiangang Hao und 5 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 68 Literaturangaben. Gefördert von: Army Research Institute for the Behavioral and Social Sciences.
Abstract (Originalsprache)
Collaborative problem solving is widely recognized as a critical 21st‐century skill. Assessing collaborative problem solving depends on coding the communication data using a construct‐relevant framework, and this process has long been a major bottleneck to scaling up such assessments. Based on five datasets and two coding frameworks, we demonstrate that ChatGPT can code communication data to a satisfactory level, though performance varies across ChatGPT models and depends on the coding framework and task characteristics. Interestingly, newer reasoning‐focused models, such as GPT‐o1‐mini and GPT‐o3‐mini, do not necessarily yield better coding results. Additionally, we show that refining prompts based on feedback from miscoded cases can improve coding accuracy in some instances, though the effectiveness of this approach is not consistent across all tasks. These findings offer practical guidance for researchers and practitioners in developing scalable, efficient methods to analyze communication data in support of 21st‐century skill assessment.
Förderung
- Army Research Institute for the Behavioral and Social Sciences
Zitieren
APA
Hao, J., Cui, W., Kyllonen, P., Kerzabi, E., Liu, L., & Flor, M. (2025). Automated Coding of Communications in Collaborative Problem‐Solving Tasks Using ChatGPT. Journal of Educational Measurement, 62(4), 809-837. https://doi.org/10.1111/jedm.70014
Vancouver
Hao J, Cui W, Kyllonen P, Kerzabi E, Liu L, Flor M. Automated Coding of Communications in Collaborative Problem‐Solving Tasks Using ChatGPT. Journal of Educational Measurement. 2025;62(4):809-837. doi:10.1111/jedm.70014
BibTeX
@article{Hao2025,
author = {Hao, Jiangang and Cui, Wenju and Kyllonen, Patrick and Kerzabi, Emily and Liu, Lei and Flor, Michael},
title = {Automated Coding of Communications in Collaborative Problem‐Solving Tasks Using ChatGPT},
journal = {Journal of Educational Measurement},
year = {2025},
volume = {62},
number = {4},
pages = {809--837},
doi = {10.1111/jedm.70014}
}
EUVIMED