EUVIMEDOpdag EUVIMED
ValidierungsstudieQuerschnittstudieOpen access

AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study

Hiromizu Takahashi iD, Kiyoshi Shikino iD, Takeshi Kondo iD, Yuji Yamada iD, Yoshitaka Tomoda iD, Minoru Kishi iD, Yuki Aiyama iD, Sho Nagai iD, Akiko Enomoto iD, Yoshinori Tokushima iD, Takahiro Shinohara iD, Fumiaki Sano iD and 3 more

JMIR Medical Education · 2026-02-17 · 12 · e81673-e81673

DOI: 10.2196/81673 ↗

At a glance

Validierungsstudie in JMIR Medical Education, published online on 2026-02-17. Written by Hiromizu Takahashi and 14 co-authors. Institution: Department of General Medicine, Faculty of Medicine, Juntendo University, 3-1-3 Hongo, Bunkyo Tokyo, 1130033 Japan, Tokyo, Japan, 81 3-3813-3111. Major MEDLINE topics: Artificial Intelligence, Clinical Competence, Interviews as Topic, Patient Simulation. Cited 9 times so far (Crossref); 19 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Generative artificial intelligence (AI) is increasingly used in medical education, including AI-based virtual patients to improve interview skills. However, how much AI-based assessment (ABA) differs from human-based assessment (HBA) remains unclear.

Objective

This study aimed to compare the quality of clinical interview assessments generated via an ABA (GPT-o1 Pro [ABA-o1] and GPT-5 Pro [ABA-5]) with those generated via an HBA conducted by clinical instructors in an AI-based virtual patient setting. We also examined whether AI reduced evaluation time and assessed agreement across participants with different levels of clinical experience.

Methods

A standardized case of leg weakness was implemented in an AI-based virtual patient. Seven participants (2 medical students, 3 residents, and 2 attending physicians) each conducted an interview with the AI patient, and transcripts were scored using the 25-item Master Interview Rating Scale (0‐125). Three evaluation strategies were compared. First, GPT-o1 Pro and GPT-5 Pro scored each transcript 5 times with different random seeds to test case specificity. Processing time was logged automatically. Second, 5 blinded clinical instructors independently rated each transcript once using the same rubric. Third, reliability metrics were applied. For AI, intraclass correlation coefficients (ICCs) quantified repeatability. For humans, the ICC(2,1) was calculated. Agreement was quantified using the Pearson r , Lin concordance correlation coefficient, Bland-Altman limits of agreement, Cronbach α, and ICC. Time efficiency was expressed as mean minutes per transcript and relative percentage reduction.

Results

Mean interview scores were similar across methods (ABA-o1: mean 52.1, SD 6.9; ABA-5: mean 53.2, SD 6.8; HBA: mean 53.7, SD 6.8). Agreement between ABA and HBA was strong ( r =0.90; concordance correlation coefficient=0.88) with minimal bias (ABA-o1: mean 0.4, SD 2.7; ABA-5: mean 1.5, SD 5.2; limits of agreement: –4.9 to 5.7 for ABA-o1 and –8.6 to 11.7 for ABA-5). The Cronbach α was 0.81 (ABA-o1), 0.86 (ABA-5), and 0.80 (HBA); the ICC(3,1) was 0.77 (ABA-o1) and 0.82 (ABA-5); and the ICC(2,1) was 0.38 (HBA). The coefficient of variation for ABA was approximately half that of HBA (6.6% vs 13.9%). Processing time for 5 runs was 4 minutes, 19 seconds for ABA-o1 and 3 minutes, 20 seconds for ABA-5 vs 10 minutes, 16 seconds for physicians, corresponding to 58% and 67.6% reductions, respectively.

Conclusions

ABA-o1 and ABA-5 produced scores closely matching HBA while demonstrating superior consistency and reliability. In the setting of virtual interview transcripts, these findings suggest that ABA may serve as a valid, rapid, and scalable alternative to HBA, reducing per-assessment time by over half. Applied strategically, AI-based scoring could enable timely feedback, improve efficiency, and reduce faculty workload. Further research is needed to confirm generalizability across broader settings.

Subjects

MeSH terms (MEDLINE)

Artificial IntelligenceClinical CompetenceInterviews as TopicPatient SimulationEducational MeasurementHumansCross-Sectional StudiesGenerative Artificial IntelligenceMaleFemaleAdult

Author keywords

AIChatGPTartificial intelligenceclinical interviewmedical educationsimulation-based learningvirtual patient

Authors and institutions

  • Department of General Medicine, Faculty of Medicine, Juntendo University, 3-1-3 Hongo, Bunkyo Tokyo, 1130033 Japan, Tokyo, Japan, 81 3-3813-3111.
  • Department of Community-Oriented Medical Education, Graduate School of Medicine, Chiba University, Chiba, Japan.
  • Center for Postgraduate Clinical Training and Career Development, Nagoya University Hospital, Nagoya, Japan.
  • The School of Health Professions Education, Maastricht University, Maastricht, The Netherlands.
  • Brookdale Department of Geriatrics and Palliative Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.
  • Department of General Internal Medicine, Itabashi Chuo Medical Center, Tokyo, Japan.

ORCID profiles: Hiromizu Takahashi, Kiyoshi Shikino, Takeshi Kondo, Yuji Yamada, Yoshitaka Tomoda, Minoru Kishi, Yuki Aiyama, Sho Nagai, Akiko Enomoto, Yoshinori Tokushima, Takahiro Shinohara, Fumiaki Sano, Takeshi Matsuura, Rikiya Watanabe, Toshio Naito

Cite

APA

Takahashi, H., Shikino, K., Kondo, T., Yamada, Y., Tomoda, Y., Kishi, M., Aiyama, Y., Nagai, S., Enomoto, A., Tokushima, Y., Shinohara, T., Sano, F., Matsuura, T., Watanabe, R., & Naito, T. (2026). AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study. JMIR Medical Education, 12, e81673-e81673. https://doi.org/10.2196/81673

Vancouver

Takahashi H, Shikino K, Kondo T, Yamada Y, Tomoda Y, Kishi M, et al. AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study. JMIR Medical Education. 2026;12:e81673-e81673. doi:10.2196/81673

BibTeX
@article{Takahashi2026,
  author = {Takahashi, Hiromizu and Shikino, Kiyoshi and Kondo, Takeshi and Yamada, Yuji and Tomoda, Yoshitaka and Kishi, Minoru and Aiyama, Yuki and Nagai, Sho and Enomoto, Akiko and Tokushima, Yoshinori and Shinohara, Takahiro and Sano, Fumiaki and Matsuura, Takeshi and Watanabe, Rikiya and Naito, Toshio},
  title = {AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e81673--e81673},
  doi = {10.2196/81673}
}
Start for free →