What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education
JMIR Medical Education · 03.09.2026 · 12 · e96819-e96819
DOI: 10.2196/96819 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Medical Education, online erschienen am 03.09.2026. Verfasst von Hui Zhang und 10 weiteren Autor:innen. Beteiligte Einrichtung: Office of Academic Affairs, Guangzhou Medical University, Guangzhou, China. Hauptthemen laut MEDLINE: Teaching, Educational Measurement, Education, Medical, Artificial Intelligence. Die Arbeit nennt 28 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language model (LLM)–based AI teaching agents are increasingly used in medical education, yet their pedagogical quality is typically judged by platform-generated scores whose scoring criteria are undisclosed and may not reflect the teaching quality of the agent.
Objective
This study aimed to develop and validate a multidimensional rubric for evaluating AI teaching agents and to examine the correspondence between platform scores and rubric-based teaching quality.
Methods
Eight AI teaching agents covering an endocrinology curriculum were deployed across 4 role-play paradigms (patient, student, expert, and family). Twenty-two fourth-year medical students generated 167 dialogues, which were scored both by the platform and by an independently applied 8-dimension rubric (100 points, covering knowledge accuracy, pedagogical guidance, knowledge coverage, role-play quality, difficulty calibration, medical safety, student engagement, and feedback quality). Each dialogue was scored 4 times by a primary evaluator (Claude Opus 4.8; mean within-model SD 0.36), with 2 additional LLMs as robustness checks; 40 dialogues spanning all agents were rescored by a medical-education expert for validation.
Results
Platform and rubric rankings diverged for most agents: the agent ranked third by the platform ranked last on rubric-based quality, and the platform’s fourth-ranked agent ranked first. Agents differed most on knowledge-related dimensions (knowledge coverage coefficient of variation=27.3%) and least on role-play quality (coefficient of variation=5.7%), while difficulty calibration was a shared weakness. In a case-level observation, one agent revised specifically to strengthen empathy attained high role-play quality yet the lowest knowledge coverage of all agents. AI scores agreed with expert ratings at the total-score level (intraclass correlation coefficient=0.51) and on cognitive-process dimensions, but agreement was low for the more subjective dimensions. Student gender showed no detectable effect, though this analysis was underpowered.
Conclusions
In this exploratory study, platform-generated scores reflected a construct different from agent teaching quality and should be used as a complement rather than as the sole quality indicator. The 8-dimension rubric provides a transparent, standardized alternative that reveals differences missed by platform scores, including a lack of association between empathy and knowledge coverage that warrants attention in future agent design.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Office of Academic Affairs, Guangzhou Medical University, Guangzhou, China.
- Department of Physiology, School of Basic Medical Sciences, The Fourth Affiliated Hospital of Guangzhou Medical University, Guangzhou, China.
- Guangzhou Municipal and Guangdong Provincial Key Laboratory of Protein Modification and Disease, Department of Physiology, School of Basic Medical Sciences, Guangzhou Medical University, 1 Xinzao Road, Guangzhou, Guangd…
- Department of Endocrinology, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China.
- Zhongshan School of Medicine, Sun Yat-Sen University, Guangzhou, China.
ORCID-Profile: Hui Zhang, Lihui Qu, Jianmin Zheng, Yi Xiong, Hongbo Bai, Ruiying Ji, Guohui Liu, Wanling Chen, Zirui Cheng, Youbang Chen, Chun-Tao Yang
Zitieren
APA
Zhang, H., Qu, L., Zheng, J., Xiong, Y., Bai, H., Ji, R., Liu, G., Chen, W., Cheng, Z., Chen, Y., & Yang, C. T. (2026). What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education. JMIR Medical Education, 12, e96819-e96819. https://doi.org/10.2196/96819
Vancouver
Zhang H, Qu L, Zheng J, Xiong Y, Bai H, Ji R, et al. What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education. JMIR Medical Education. 2026;12:e96819-e96819. doi:10.2196/96819
BibTeX
@article{Zhang2026,
author = {Zhang, Hui and Qu, Lihui and Zheng, Jianmin and Xiong, Yi and Bai, Hongbo and Ji, Ruiying and Liu, Guohui and Chen, Wanling and Cheng, Zirui and Chen, Youbang and Yang, Chun-Tao},
title = {What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education},
journal = {JMIR Medical Education},
year = {2026},
volume = {12},
pages = {e96819--e96819},
doi = {10.2196/96819}
}
EUVIMED