EUVIMEDScopri EUVIMED
Open access

AI-Driven Objective Structured Clinical Examination Generation in Digital Health Education: Comparative Analysis of Three GPT-4o Configurations

KI-gesteuerte Generierung von Objective Structured Clinical Examinations in der digitalen Gesundheitsbildung: vergleichende Analyse von 3 GPT-4o-Konfigurationen

Zineb Zouakia iD, Emmanuel Logak iD, Alan Szymczak iD, Jean-Philippe Jais iD, Anita Burgun iD, Rosy Tsopra iD

JMIR Medical Education · 2026-01-15 · 12 · e82116

DOI: 10.2196/82116 ↗

Auf einen Blick

Drei GPT-4o-Konfigurationen zur OSCE-Generierung wurden verglichen. Die simulierte-Agenten-Konfiguration schnitt bei Format- und Inhaltsqualität am besten ab.

Fragestellung

Wie unterscheiden sich 3 GPT-4o-Konfigurationen bei der Generierung von OSCE-Stationen für digitale Gesundheit?

Methode

Vergleichende Analyse von 3 GPT-4o-Konfigurationen; 24 OSCE-Stationen zu 8 digitalen Gesundheitsthemen; Bewertung durch 1 Expert:in und 2 verblindete Expert:innen; Kruskal-Wallis-Tests.

Ergebnisse

Simulated-agents GPT erzielte die besten Werte bei Formatkonformität und mehreren Inhaltskriterien, darunter Genauigkeit (Mittelwert 4,47/5; SD 0,28; P=.01) und Klarheit (Mittelwert 4,46/5; SD 0,52; P=.004). 88 % (14/16) waren ohne größere Überarbeitungen nutzbar.

Schlussfolgerung der Autor:innen

Strukturierte Prompting-Strategien, besonders die Simulation von Agenten, verbessern laut Autor:innen Zuverlässigkeit und Nutzbarkeit von LLM-generierten OSCE-Inhalten. Die Ergebnisse unterstützen den Einsatz von KI in der medizinischen Ausbildung bei weiterhin nötiger Expertenvalidierung.

Kernaussagen

  • Simulated-agents GPT war in der Studie am besten bewertet.
  • Genauigkeit: Mittelwert 4,47/5; Klarheit: Mittelwert 4,46/5.
  • Die Autor:innen betonen die nötige Expertenvalidierung.

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Journal article in JMIR Medical Education, published online on 2026-01-15. Written by Zineb Zouakia and 5 co-authors. Institution: Clinical Bioinformatics Laboratory, Imagine Institute, Université Paris Cité, INSERM UMR1163, Paris, France. Major MEDLINE topics: Educational Measurement, Artificial Intelligence. Cited 8 times so far (Crossref); 74 references. The full text is freely available in PubMed Central.

Auf einen BlickAbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Background

Objective Structured Clinical Examinations (OSCEs) are used as an evaluation method in medical education, but require significant pedagogical expertise and investment, especially in emerging fields like digital health. Large language models (LLMs), such as ChatGPT (OpenAI), have shown potential in automating educational content generation. However, OSCE generation using LLMs remains underexplored.

Objective

This study aims to evaluate 3 GPT-4o configurations for generating OSCE stations in digital health: (1) standard GPT with a simple prompt and OSCE guidelines; (2) personalized GPT with a simple prompt, OSCE guidelines, and a reference book in digital health; and (3) simulated-agents GPT with a structured prompt simulating specialized OSCE agents and the digital health reference book.

Methods

Overall, 24 OSCE stations were generated across 8 digital health topics with each GPT-4o configuration. Format compliance was evaluated by one expert, while educational content was assessed independently by 2 digital health experts, blind to GPT-4o configurations, using a comprehensive assessment grid. Statistical analyses were performed using Kruskal-Wallis tests.

Results

Simulated-agents GPT performed best in format compliance and most content quality criteria, including accuracy (mean 4.47/5, SD 0.28; P=.01) and clarity (mean 4.46/5, SD 0.52; P=.004). It also had 88% (14/16) for usability without major revisions and first-place preference ranking, outperforming the other configurations. Personalized GPT showed the lowest format compliance, while standard GPT scored lowest for clarity and educational value.

Conclusions

Structured prompting strategies, particularly agents’ simulation, enhance the reliability and usability of LLM-generated OSCE content. These results support the use of artificial intelligence in medical education, while confirming the need for expert validation.

Subjects

MeSH terms (MEDLINE)

Educational MeasurementArtificial IntelligenceLarge Language ModelsHumansDigital HealthGenerative Artificial Intelligence

Author keywords

ChatGPTGPT-4odigital healthdigital health educationgenerative artificial intelligencelarge language modelsmedical educationmedical informaticsobjective structured clinical examinationprompt design

Authors and institutions

  • Clinical Bioinformatics Laboratory, Imagine Institute, Université Paris Cité, INSERM UMR1163, Paris, France.
  • Department of Medical Informatics, Hôpital Européen Georges Pompidou, Hôpital Necker Enfants Malades, AP-HP, Paris, France.
  • Imagine Institute, Université Paris Cité, Paris, France.
  • Biostatistic Unit, Necker University Hospital, AP-HP, Paris, France.
  • UPPERS US 007 F-75006, Université Paris Cité, Paris, France.

ORCID profiles: Zineb Zouakia, Emmanuel Logak, Alan Szymczak, Jean-Philippe Jais, Anita Burgun, Rosy Tsopra

Cite

APA

Zouakia, Z., Logak, E., Szymczak, A., Jais, J. P., Burgun, A., & Tsopra, R. (2026). AI-Driven Objective Structured Clinical Examination Generation in Digital Health Education: Comparative Analysis of Three GPT-4o Configurations. JMIR Medical Education, 12, e82116. https://doi.org/10.2196/82116

Vancouver

Zouakia Z, Logak E, Szymczak A, Jais JP, Burgun A, Tsopra R. AI-Driven Objective Structured Clinical Examination Generation in Digital Health Education: Comparative Analysis of Three GPT-4o Configurations. JMIR Medical Education. 2026;12:e82116. doi:10.2196/82116

BibTeX
@article{Zouakia2026,
  author = {Zouakia, Zineb and Logak, Emmanuel and Szymczak, Alan and Jais, Jean-Philippe and Burgun, Anita and Tsopra, Rosy},
  title = {AI-Driven Objective Structured Clinical Examination Generation in Digital Health Education: Comparative Analysis of Three GPT-4o Configurations},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e82116},
  doi = {10.2196/82116}
}
Start for free →