EUVIMEDEUVIMED entdecken
QuerschnittstudieOpen Access

Evaluating Large Language Model–Based Automated Scoring in a Voice-Based Virtual Standardized Patient Platform for Medical Students: Cross-Sectional Agreement Study

Xiaoxing Gao iD, Xiaoming Huang iD, Rongrong Hu iD, Li Zhang iD, Huiting Liu iD, Bingqing Zhang iD, Chong Wei iD, Wei Qiu iD, Mengyu Zhang iD, Xuefeng Sun iD

JMIR Medical Education · 24.09.2026 · 12 · e95578-e95578

DOI: 10.2196/95578 ↗

Kurz gefasst

Querschnittstudie in der Zeitschrift JMIR Medical Education, online erschienen am 24.09.2026. Verfasst von Xiaoxing Gao und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Pulmonary and Critical Care Medicine, Peking Union Medical College Hospital, 1 Shuaifuyuan, Dongcheng District, Beijing, 100730, China,…. Hauptthemen laut MEDLINE: Educational Measurement, Students, Medical, Patient Simulation. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language model (LLM)–powered virtual standardized patients (VSPs) enable scalable clinical skills practice, but the validity of AI-generated scores relative to faculty ratings remains unclear.

Objective

This study aimed to assess agreement between LLM-generated and faculty ratings of history-taking and communication performance and to examine the influence of rater and case heterogeneity.

Methods

In this cross-sectional study, 92 fourth-year medical students completed one of three 15-minute voice-based VSP cases (fever, diarrhea, and cough). Ten blinded faculty raters scored performance (0‐100 points total; 0‐50 points per domain). AI scores were generated by DeepSeek-V3 using a calibrated prompt. Agreement was evaluated using mixed-effects models, intraclass correlation coefficients (ICC [2,1]), Spearman correlations, mean absolute error (MAE), Bland-Altman analysis, and variance partition coefficients (VPC).

Results

Median total scores were similar for AI and faculty (median 93.0, IQR 89.0-95.0 vs median 94.0, IQR 91.0-95.0). Rater variability accounted for 37% of residual variance in faculty total scores (VPC=0.37). AI total scores were positively associated with faculty total scores (β=0.37, 95% CI 0.26‐0.48; P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Educational MeasurementStudents, MedicalPatient SimulationCross-Sectional StudiesHumansLarge Language ModelsFemaleMaleClinical CompetenceReproducibility of Results

Stichwörter der Autor:innen

communication skillsinformation gatheringlarge language modelmixed-effects modelrater variabilityvirtual standardized patient

Autor:innen und Institutionen

  • Department of Pulmonary and Critical Care Medicine, Peking Union Medical College Hospital, 1 Shuaifuyuan, Dongcheng District, Beijing, 100730, China, 86 10 6915 5000.
  • Department of General Internal Medicine, Peking Union Medical College Hospital, Beijing, China.
  • Department of Nephrology, Peking Union Medical College Hospital, Beijing, China.
  • Department of Rheumatology, Peking Union Medical College Hospital, Beijing, China.
  • Department of Infectious Disease, Peking Union Medical College Hospital, Beijing, China.
  • Department of Haematology, Peking Union Medical College Hospital, Beijing, China.

ORCID-Profile: Xiaoxing Gao, Xiaoming Huang, Rongrong Hu, Li Zhang, Huiting Liu, Bingqing Zhang, Chong Wei, Wei Qiu, Mengyu Zhang, Xuefeng Sun

Zitieren

APA

Gao, X., Huang, X., Hu, R., Zhang, L., Liu, H., Zhang, B., Wei, C., Qiu, W., Zhang, M., & Sun, X. (2026). Evaluating Large Language Model–Based Automated Scoring in a Voice-Based Virtual Standardized Patient Platform for Medical Students: Cross-Sectional Agreement Study. JMIR Medical Education, 12, e95578-e95578. https://doi.org/10.2196/95578

Vancouver

Gao X, Huang X, Hu R, Zhang L, Liu H, Zhang B, et al. Evaluating Large Language Model–Based Automated Scoring in a Voice-Based Virtual Standardized Patient Platform for Medical Students: Cross-Sectional Agreement Study. JMIR Medical Education. 2026;12:e95578-e95578. doi:10.2196/95578

BibTeX
@article{Gao2026,
  author = {Gao, Xiaoxing and Huang, Xiaoming and Hu, Rongrong and Zhang, Li and Liu, Huiting and Zhang, Bingqing and Wei, Chong and Qiu, Wei and Zhang, Mengyu and Sun, Xuefeng},
  title = {Evaluating Large Language Model–Based Automated Scoring in a Voice-Based Virtual Standardized Patient Platform for Medical Students: Cross-Sectional Agreement Study},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e95578--e95578},
  doi = {10.2196/95578}
}
Kostenlos starten →