EUVIMEDEUVIMED entdecken
Open Access

What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education

Hui Zhang iD, Lihui Qu iD, Jianmin Zheng iD, Yi Xiong iD, Hongbo Bai iD, Ruiying Ji iD, Guohui Liu iD, Wanling Chen iD, Zirui Cheng iD, Youbang Chen iD, Chun-Tao Yang iD

JMIR Medical Education · 03.09.2026 · 12 · e96819-e96819

DOI: 10.2196/96819 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Education, online erschienen am 03.09.2026. Verfasst von Hui Zhang und 10 weiteren Autor:innen. Beteiligte Einrichtung: Office of Academic Affairs, Guangzhou Medical University, Guangzhou, China. Hauptthemen laut MEDLINE: Teaching, Educational Measurement, Education, Medical, Artificial Intelligence. Die Arbeit nennt 28 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language model (LLM)–based AI teaching agents are increasingly used in medical education, yet their pedagogical quality is typically judged by platform-generated scores whose scoring criteria are undisclosed and may not reflect the teaching quality of the agent.

Objective

This study aimed to develop and validate a multidimensional rubric for evaluating AI teaching agents and to examine the correspondence between platform scores and rubric-based teaching quality.

Methods

Eight AI teaching agents covering an endocrinology curriculum were deployed across 4 role-play paradigms (patient, student, expert, and family). Twenty-two fourth-year medical students generated 167 dialogues, which were scored both by the platform and by an independently applied 8-dimension rubric (100 points, covering knowledge accuracy, pedagogical guidance, knowledge coverage, role-play quality, difficulty calibration, medical safety, student engagement, and feedback quality). Each dialogue was scored 4 times by a primary evaluator (Claude Opus 4.8; mean within-model SD 0.36), with 2 additional LLMs as robustness checks; 40 dialogues spanning all agents were rescored by a medical-education expert for validation.

Results

Platform and rubric rankings diverged for most agents: the agent ranked third by the platform ranked last on rubric-based quality, and the platform’s fourth-ranked agent ranked first. Agents differed most on knowledge-related dimensions (knowledge coverage coefficient of variation=27.3%) and least on role-play quality (coefficient of variation=5.7%), while difficulty calibration was a shared weakness. In a case-level observation, one agent revised specifically to strengthen empathy attained high role-play quality yet the lowest knowledge coverage of all agents. AI scores agreed with expert ratings at the total-score level (intraclass correlation coefficient=0.51) and on cognitive-process dimensions, but agreement was low for the more subjective dimensions. Student gender showed no detectable effect, though this analysis was underpowered.

Conclusions

In this exploratory study, platform-generated scores reflected a construct different from agent teaching quality and should be used as a complement rather than as the sole quality indicator. The 8-dimension rubric provides a transparent, standardized alternative that reveals differences missed by platform scores, including a lack of association between empathy and knowledge coverage that warrants attention in future agent design.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

TeachingEducational MeasurementEducation, MedicalArtificial IntelligenceHumansLarge Language ModelsCurriculumStudents, Medical

Stichwörter der Autor:innen

AI teaching agentsLLMseducational data scienceeducational evaluationendocrinologylarge language modelsmultidimensional rubricprecision educationrole-play

Autor:innen und Institutionen

  • Office of Academic Affairs, Guangzhou Medical University, Guangzhou, China.
  • Department of Physiology, School of Basic Medical Sciences, The Fourth Affiliated Hospital of Guangzhou Medical University, Guangzhou, China.
  • Guangzhou Municipal and Guangdong Provincial Key Laboratory of Protein Modification and Disease, Department of Physiology, School of Basic Medical Sciences, Guangzhou Medical University, 1 Xinzao Road, Guangzhou, Guangd…
  • Department of Endocrinology, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China.
  • Zhongshan School of Medicine, Sun Yat-Sen University, Guangzhou, China.

ORCID-Profile: Hui Zhang, Lihui Qu, Jianmin Zheng, Yi Xiong, Hongbo Bai, Ruiying Ji, Guohui Liu, Wanling Chen, Zirui Cheng, Youbang Chen, Chun-Tao Yang

Zitieren

APA

Zhang, H., Qu, L., Zheng, J., Xiong, Y., Bai, H., Ji, R., Liu, G., Chen, W., Cheng, Z., Chen, Y., & Yang, C. T. (2026). What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education. JMIR Medical Education, 12, e96819-e96819. https://doi.org/10.2196/96819

Vancouver

Zhang H, Qu L, Zheng J, Xiong Y, Bai H, Ji R, et al. What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education. JMIR Medical Education. 2026;12:e96819-e96819. doi:10.2196/96819

BibTeX
@article{Zhang2026,
  author = {Zhang, Hui and Qu, Lihui and Zheng, Jianmin and Xiong, Yi and Bai, Hongbo and Ji, Ruiying and Liu, Guohui and Chen, Wanling and Cheng, Zirui and Chen, Youbang and Yang, Chun-Tao},
  title = {What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e96819--e96819},
  doi = {10.2196/96819}
}
Kostenlos starten →