EUVIMEDEUVIMED entdecken
ValidierungsstudieQuerschnittstudieOpen Access

AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study

Hiromizu Takahashi iD, Kiyoshi Shikino iD, Takeshi Kondo iD, Yuji Yamada iD, Yoshitaka Tomoda iD, Minoru Kishi iD, Yuki Aiyama iD, Sho Nagai iD, Akiko Enomoto iD, Yoshinori Tokushima iD, Takahiro Shinohara iD, Fumiaki Sano iD und 3 weitere

JMIR Medical Education · 17.02.2026 · 12 · e81673-e81673

DOI: 10.2196/81673 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift JMIR Medical Education, online erschienen am 17.02.2026. Verfasst von Hiromizu Takahashi und 14 weiteren Autor:innen. Beteiligte Einrichtung: Department of General Medicine, Faculty of Medicine, Juntendo University, 3-1-3 Hongo, Bunkyo Tokyo, 1130033 Japan, Tokyo, Japan, 81 3-3813-3111. Hauptthemen laut MEDLINE: Artificial Intelligence, Clinical Competence, Interviews as Topic, Patient Simulation. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Generative artificial intelligence (AI) is increasingly used in medical education, including AI-based virtual patients to improve interview skills. However, how much AI-based assessment (ABA) differs from human-based assessment (HBA) remains unclear.

Objective

This study aimed to compare the quality of clinical interview assessments generated via an ABA (GPT-o1 Pro [ABA-o1] and GPT-5 Pro [ABA-5]) with those generated via an HBA conducted by clinical instructors in an AI-based virtual patient setting. We also examined whether AI reduced evaluation time and assessed agreement across participants with different levels of clinical experience.

Methods

A standardized case of leg weakness was implemented in an AI-based virtual patient. Seven participants (2 medical students, 3 residents, and 2 attending physicians) each conducted an interview with the AI patient, and transcripts were scored using the 25-item Master Interview Rating Scale (0‐125). Three evaluation strategies were compared. First, GPT-o1 Pro and GPT-5 Pro scored each transcript 5 times with different random seeds to test case specificity. Processing time was logged automatically. Second, 5 blinded clinical instructors independently rated each transcript once using the same rubric. Third, reliability metrics were applied. For AI, intraclass correlation coefficients (ICCs) quantified repeatability. For humans, the ICC(2,1) was calculated. Agreement was quantified using the Pearson r , Lin concordance correlation coefficient, Bland-Altman limits of agreement, Cronbach α, and ICC. Time efficiency was expressed as mean minutes per transcript and relative percentage reduction.

Results

Mean interview scores were similar across methods (ABA-o1: mean 52.1, SD 6.9; ABA-5: mean 53.2, SD 6.8; HBA: mean 53.7, SD 6.8). Agreement between ABA and HBA was strong ( r =0.90; concordance correlation coefficient=0.88) with minimal bias (ABA-o1: mean 0.4, SD 2.7; ABA-5: mean 1.5, SD 5.2; limits of agreement: –4.9 to 5.7 for ABA-o1 and –8.6 to 11.7 for ABA-5). The Cronbach α was 0.81 (ABA-o1), 0.86 (ABA-5), and 0.80 (HBA); the ICC(3,1) was 0.77 (ABA-o1) and 0.82 (ABA-5); and the ICC(2,1) was 0.38 (HBA). The coefficient of variation for ABA was approximately half that of HBA (6.6% vs 13.9%). Processing time for 5 runs was 4 minutes, 19 seconds for ABA-o1 and 3 minutes, 20 seconds for ABA-5 vs 10 minutes, 16 seconds for physicians, corresponding to 58% and 67.6% reductions, respectively.

Conclusions

ABA-o1 and ABA-5 produced scores closely matching HBA while demonstrating superior consistency and reliability. In the setting of virtual interview transcripts, these findings suggest that ABA may serve as a valid, rapid, and scalable alternative to HBA, reducing per-assessment time by over half. Applied strategically, AI-based scoring could enable timely feedback, improve efficiency, and reduce faculty workload. Further research is needed to confirm generalizability across broader settings.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceClinical CompetenceInterviews as TopicPatient SimulationEducational MeasurementHumansCross-Sectional StudiesGenerative Artificial IntelligenceMaleFemaleAdult

Stichwörter der Autor:innen

AIChatGPTartificial intelligenceclinical interviewmedical educationsimulation-based learningvirtual patient

Autor:innen und Institutionen

  • Department of General Medicine, Faculty of Medicine, Juntendo University, 3-1-3 Hongo, Bunkyo Tokyo, 1130033 Japan, Tokyo, Japan, 81 3-3813-3111.
  • Department of Community-Oriented Medical Education, Graduate School of Medicine, Chiba University, Chiba, Japan.
  • Center for Postgraduate Clinical Training and Career Development, Nagoya University Hospital, Nagoya, Japan.
  • The School of Health Professions Education, Maastricht University, Maastricht, The Netherlands.
  • Brookdale Department of Geriatrics and Palliative Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.
  • Department of General Internal Medicine, Itabashi Chuo Medical Center, Tokyo, Japan.

ORCID-Profile: Hiromizu Takahashi, Kiyoshi Shikino, Takeshi Kondo, Yuji Yamada, Yoshitaka Tomoda, Minoru Kishi, Yuki Aiyama, Sho Nagai, Akiko Enomoto, Yoshinori Tokushima, Takahiro Shinohara, Fumiaki Sano, Takeshi Matsuura, Rikiya Watanabe, Toshio Naito

Zitieren

APA

Takahashi, H., Shikino, K., Kondo, T., Yamada, Y., Tomoda, Y., Kishi, M., Aiyama, Y., Nagai, S., Enomoto, A., Tokushima, Y., Shinohara, T., Sano, F., Matsuura, T., Watanabe, R., & Naito, T. (2026). AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study. JMIR Medical Education, 12, e81673-e81673. https://doi.org/10.2196/81673

Vancouver

Takahashi H, Shikino K, Kondo T, Yamada Y, Tomoda Y, Kishi M, et al. AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study. JMIR Medical Education. 2026;12:e81673-e81673. doi:10.2196/81673

BibTeX
@article{Takahashi2026,
  author = {Takahashi, Hiromizu and Shikino, Kiyoshi and Kondo, Takeshi and Yamada, Yuji and Tomoda, Yoshitaka and Kishi, Minoru and Aiyama, Yuki and Nagai, Sho and Enomoto, Akiko and Tokushima, Yoshinori and Shinohara, Takahiro and Sano, Fumiaki and Matsuura, Takeshi and Watanabe, Rikiya and Naito, Toshio},
  title = {AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e81673--e81673},
  doi = {10.2196/81673}
}
Kostenlos starten →