AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study
JMIR Medical Education · 17.02.2026 · 12 · e81673-e81673
DOI: 10.2196/81673 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift JMIR Medical Education, online erschienen am 17.02.2026. Verfasst von Hiromizu Takahashi und 14 weiteren Autor:innen. Beteiligte Einrichtung: Department of General Medicine, Faculty of Medicine, Juntendo University, 3-1-3 Hongo, Bunkyo Tokyo, 1130033 Japan, Tokyo, Japan, 81 3-3813-3111. Hauptthemen laut MEDLINE: Artificial Intelligence, Clinical Competence, Interviews as Topic, Patient Simulation. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Generative artificial intelligence (AI) is increasingly used in medical education, including AI-based virtual patients to improve interview skills. However, how much AI-based assessment (ABA) differs from human-based assessment (HBA) remains unclear.
Objective
This study aimed to compare the quality of clinical interview assessments generated via an ABA (GPT-o1 Pro [ABA-o1] and GPT-5 Pro [ABA-5]) with those generated via an HBA conducted by clinical instructors in an AI-based virtual patient setting. We also examined whether AI reduced evaluation time and assessed agreement across participants with different levels of clinical experience.
Methods
A standardized case of leg weakness was implemented in an AI-based virtual patient. Seven participants (2 medical students, 3 residents, and 2 attending physicians) each conducted an interview with the AI patient, and transcripts were scored using the 25-item Master Interview Rating Scale (0‐125). Three evaluation strategies were compared. First, GPT-o1 Pro and GPT-5 Pro scored each transcript 5 times with different random seeds to test case specificity. Processing time was logged automatically. Second, 5 blinded clinical instructors independently rated each transcript once using the same rubric. Third, reliability metrics were applied. For AI, intraclass correlation coefficients (ICCs) quantified repeatability. For humans, the ICC(2,1) was calculated. Agreement was quantified using the Pearson r , Lin concordance correlation coefficient, Bland-Altman limits of agreement, Cronbach α, and ICC. Time efficiency was expressed as mean minutes per transcript and relative percentage reduction.
Results
Mean interview scores were similar across methods (ABA-o1: mean 52.1, SD 6.9; ABA-5: mean 53.2, SD 6.8; HBA: mean 53.7, SD 6.8). Agreement between ABA and HBA was strong ( r =0.90; concordance correlation coefficient=0.88) with minimal bias (ABA-o1: mean 0.4, SD 2.7; ABA-5: mean 1.5, SD 5.2; limits of agreement: –4.9 to 5.7 for ABA-o1 and –8.6 to 11.7 for ABA-5). The Cronbach α was 0.81 (ABA-o1), 0.86 (ABA-5), and 0.80 (HBA); the ICC(3,1) was 0.77 (ABA-o1) and 0.82 (ABA-5); and the ICC(2,1) was 0.38 (HBA). The coefficient of variation for ABA was approximately half that of HBA (6.6% vs 13.9%). Processing time for 5 runs was 4 minutes, 19 seconds for ABA-o1 and 3 minutes, 20 seconds for ABA-5 vs 10 minutes, 16 seconds for physicians, corresponding to 58% and 67.6% reductions, respectively.
Conclusions
ABA-o1 and ABA-5 produced scores closely matching HBA while demonstrating superior consistency and reliability. In the setting of virtual interview transcripts, these findings suggest that ABA may serve as a valid, rapid, and scalable alternative to HBA, reducing per-assessment time by over half. Applied strategically, AI-based scoring could enable timely feedback, improve efficiency, and reduce faculty workload. Further research is needed to confirm generalizability across broader settings.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of General Medicine, Faculty of Medicine, Juntendo University, 3-1-3 Hongo, Bunkyo Tokyo, 1130033 Japan, Tokyo, Japan, 81 3-3813-3111.
- Department of Community-Oriented Medical Education, Graduate School of Medicine, Chiba University, Chiba, Japan.
- Center for Postgraduate Clinical Training and Career Development, Nagoya University Hospital, Nagoya, Japan.
- The School of Health Professions Education, Maastricht University, Maastricht, The Netherlands.
- Brookdale Department of Geriatrics and Palliative Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.
- Department of General Internal Medicine, Itabashi Chuo Medical Center, Tokyo, Japan.
ORCID-Profile: Hiromizu Takahashi, Kiyoshi Shikino, Takeshi Kondo, Yuji Yamada, Yoshitaka Tomoda, Minoru Kishi, Yuki Aiyama, Sho Nagai, Akiko Enomoto, Yoshinori Tokushima, Takahiro Shinohara, Fumiaki Sano, Takeshi Matsuura, Rikiya Watanabe, Toshio Naito
Zitieren
APA
Takahashi, H., Shikino, K., Kondo, T., Yamada, Y., Tomoda, Y., Kishi, M., Aiyama, Y., Nagai, S., Enomoto, A., Tokushima, Y., Shinohara, T., Sano, F., Matsuura, T., Watanabe, R., & Naito, T. (2026). AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study. JMIR Medical Education, 12, e81673-e81673. https://doi.org/10.2196/81673
Vancouver
Takahashi H, Shikino K, Kondo T, Yamada Y, Tomoda Y, Kishi M, et al. AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study. JMIR Medical Education. 2026;12:e81673-e81673. doi:10.2196/81673
BibTeX
@article{Takahashi2026,
author = {Takahashi, Hiromizu and Shikino, Kiyoshi and Kondo, Takeshi and Yamada, Yuji and Tomoda, Yoshitaka and Kishi, Minoru and Aiyama, Yuki and Nagai, Sho and Enomoto, Akiko and Tokushima, Yoshinori and Shinohara, Takahiro and Sano, Fumiaki and Matsuura, Takeshi and Watanabe, Rikiya and Naito, Toshio},
title = {AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study},
journal = {JMIR Medical Education},
year = {2026},
volume = {12},
pages = {e81673--e81673},
doi = {10.2196/81673}
}
EUVIMED