EUVIMEDEUVIMED entdecken
Crossref

Automated Coding of Communication Data Using LLM: Consistency across Subgroups

Jiangang Hao, Wenju Cui, Patrick Kyllonen, Emily Kerzabi

Journal of Educational Measurement · 2026-06 · 63 (2)

DOI: 10.1111/jedm.70049 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Educational Measurement, online erschienen am 04.06.2026. Verfasst von Jiangang Hao und 3 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 53 Literaturangaben. Gefördert von: Army Research Institute for the Behavioral and Social Sciences.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Assessing communication and collaboration at scale depends on a labor‐intensive task of coding communication data into categories according to different frameworks. Prior research has established that large language models (LLMs), particularly those from the GPT family, can be directly instructed with coding rubrics to code communication data and achieve accuracy comparable to human raters. However, whether the coding from LLMs perform consistently across different demographic groups, such as gender and race, remains unclear. To address this gap, we introduce three checks for evaluating subgroup consistency of LLM‐based coding, and empirically examine them based on data from three types of collaborative tasks. Our results show that LLM‐based coding perform consistently in the same way as human raters across gender or racial/ethnic groups, demonstrating the possibility of its use in large‐scale assessments of collaboration, communication, and other AI‐enabled conversation‐based assessments.

Förderung

  • Army Research Institute for the Behavioral and Social Sciences

Zitieren

APA

Hao, J., Cui, W., Kyllonen, P., & Kerzabi, E. (2026). Automated Coding of Communication Data Using LLM: Consistency across Subgroups. Journal of Educational Measurement, 63(2). https://doi.org/10.1111/jedm.70049

Vancouver

Hao J, Cui W, Kyllonen P, Kerzabi E. Automated Coding of Communication Data Using LLM: Consistency across Subgroups. Journal of Educational Measurement. 2026;63(2). doi:10.1111/jedm.70049

BibTeX
@article{Hao2026,
  author = {Hao, Jiangang and Cui, Wenju and Kyllonen, Patrick and Kerzabi, Emily},
  title = {Automated Coding of Communication Data Using LLM: Consistency across Subgroups},
  journal = {Journal of Educational Measurement},
  year = {2026},
  volume = {63},
  number = {2},
  doi = {10.1111/jedm.70049}
}
Kostenlos starten →