A Pilot Comparability Study of GPT-4-Generated and Expert-Written OSCE Scripts
Medical Science Educator · 05.09.2026 · 36 (6) · 3169-3177
DOI: 10.1007/s40670-026-02877-3 ↗Kurz gefasst
Fachartikel in der Zeitschrift Medical Science Educator, online erschienen am 05.09.2026. Verfasst von Hirday Josan und 8 weiteren Autor:innen. Beteiligte Einrichtung: Faculty of Science, McMaster University, 1280 Main Street West, L8S 4L8, Hamilton, ON Canada. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 20 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Developing high-quality Objective Structured Clinical Examination (OSCE) standardized patient (SP) scripts is time and resource-intensive. Large language models such as GPT-4 may improve script writing, however, their educational adequacy and realism compared with expert-written scripts remain uncertain.
Objective
We compared GPT-4 generated OSCE scripts to expert-written scripts across five educational quality domains: detectability of AI authorship, perceived realism, consistency of role depiction, clinical accuracy, and alignment with learning objectives. A secondary exploratory non-inferiority analysis was also conducted.
Methods
We developed fifteen OSCE SP scripts using three approaches, expert-written, AI-generated, and AI-generated with human editing, with five scripts per group. Scripts were rated by blinded SPs and physicians. We assessed whether AI-generated scripts were non-inferior to expert-written scripts within 10-percentage-points.
Results
Raters could not reliably distinguish either AI-generated script group from expert-authored scripts. For realism, means were 3.90 (SD 0.79) for expert-written, 3.88 (1.02) for AI-generated, and 3.56 (1.07) for AI+human scripts. For clinical accuracy, means were 4.27 (0.70), 4.07 (0.96), and 4.40 (0.63). For alignment with learning objectives, they were 4.27 (0.80), 3.67 (1.11), and 4.27 (0.80). AI+human scripts were rated highest for clinical accuracy and alignment with learning objectives, and expert-written scripts highest for realism and consistency of role depiction. In the exploratory non-inferiority analysis, non-inferiority to expert-written scripts was not demonstrated for either AI-generated set.
Conclusion
This pilot study provides preliminary evidence on the comparability of GPT-4-generated and expert-written OSCE scripts, with similar ratings across several educational quality domains, though non-inferiority was not demonstrated. Larger, more rigorous studies are needed to conclusively demonstrate AI generated OSCE scripts are non-inferior when compared to human expert generated OSCE scripts.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Faculty of Science, McMaster University, 1280 Main Street West, L8S 4L8, Hamilton, ON Canada.
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON Canada.
- Toronto Metropolitan University School of Medicine, Brampton, ON Canada.
- Faculty of Health Sciences, Western University, London, ON Canada.
- Centre for Addiction & Mental Health, Toronto, ON Canada.
- Oakville Trafalgar Memorial Hospital, Oakville, ON Canada.
ORCID-Profile: Hirday Josan, Sina Sayyad, Madeleine Zanin, Daniel Sutton, Christopher Meaney, Fok-Han Leung
Zitieren
APA
Josan, H., Sayyad, S., Zanin, M., Sutton, D., Vakil, T., Mirzo, S. M., Sandu, K., Meaney, C., & Leung, F. H. (2026). A Pilot Comparability Study of GPT-4-Generated and Expert-Written OSCE Scripts. Medical Science Educator, 36(6), 3169-3177. https://doi.org/10.1007/s40670-026-02877-3
Vancouver
Josan H, Sayyad S, Zanin M, Sutton D, Vakil T, Mirzo SM, et al. A Pilot Comparability Study of GPT-4-Generated and Expert-Written OSCE Scripts. Medical Science Educator. 2026;36(6):3169-3177. doi:10.1007/s40670-026-02877-3
BibTeX
@article{Josan2026,
author = {Josan, Hirday and Sayyad, Sina and Zanin, Madeleine and Sutton, Daniel and Vakil, Tanvi and Mirzo, Suhrob Mullo and Sandu, Karanbir and Meaney, Christopher and Leung, Fok-Han},
title = {A Pilot Comparability Study of GPT-4-Generated and Expert-Written OSCE Scripts},
journal = {Medical Science Educator},
year = {2026},
volume = {36},
number = {6},
pages = {3169--3177},
doi = {10.1007/s40670-026-02877-3}
}
EUVIMED