Automated Coding of Communication Data Using LLM: Consistency across Subgroups
Journal of Educational Measurement · 2026-06 · 63 (2)
DOI: 10.1111/jedm.70049 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Educational Measurement, online erschienen am 04.06.2026. Verfasst von Jiangang Hao und 3 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 53 Literaturangaben. Gefördert von: Army Research Institute for the Behavioral and Social Sciences.
Abstract (Originalsprache)
Assessing communication and collaboration at scale depends on a labor‐intensive task of coding communication data into categories according to different frameworks. Prior research has established that large language models (LLMs), particularly those from the GPT family, can be directly instructed with coding rubrics to code communication data and achieve accuracy comparable to human raters. However, whether the coding from LLMs perform consistently across different demographic groups, such as gender and race, remains unclear. To address this gap, we introduce three checks for evaluating subgroup consistency of LLM‐based coding, and empirically examine them based on data from three types of collaborative tasks. Our results show that LLM‐based coding perform consistently in the same way as human raters across gender or racial/ethnic groups, demonstrating the possibility of its use in large‐scale assessments of collaboration, communication, and other AI‐enabled conversation‐based assessments.
Förderung
- Army Research Institute for the Behavioral and Social Sciences
Zitieren
APA
Hao, J., Cui, W., Kyllonen, P., & Kerzabi, E. (2026). Automated Coding of Communication Data Using LLM: Consistency across Subgroups. Journal of Educational Measurement, 63(2). https://doi.org/10.1111/jedm.70049
Vancouver
Hao J, Cui W, Kyllonen P, Kerzabi E. Automated Coding of Communication Data Using LLM: Consistency across Subgroups. Journal of Educational Measurement. 2026;63(2). doi:10.1111/jedm.70049
BibTeX
@article{Hao2026,
author = {Hao, Jiangang and Cui, Wenju and Kyllonen, Patrick and Kerzabi, Emily},
title = {Automated Coding of Communication Data Using LLM: Consistency across Subgroups},
journal = {Journal of Educational Measurement},
year = {2026},
volume = {63},
number = {2},
doi = {10.1111/jedm.70049}
}
EUVIMED