Performance and Hallucination Analysis of Large Language Models on European Anesthesiology Examinations: Cross-Sectional Comparative Study
JMIR Formative Research · 01.09.2026 · 10 · e95859-e95859
DOI: 10.2196/95859 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift JMIR Formative Research, online erschienen am 01.09.2026. Verfasst von Stefan Andrei und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Anesthesiology and Intensive Care, CHU Bichat Claude Bernard, Assistance Publique, Hôpitaux de Paris, Paris, Île-de-France, France. Hauptthemen laut MEDLINE: Large Language Models, Anesthesiology, Educational Measurement, Hallucinations. Die Arbeit nennt 30 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models (LLMs) have shown promising performance on medical examinations across specialties. However, comparative evaluations of current-generation LLMs across multiple European anesthesiology examinations, alongside structured assessment of hallucinations vs question-related confusion, remain lacking.
Objective
This study aimed to compare the performance of 4 state-of-the-art LLMs on anesthesiology and intensive medicine examination questions and assess their hallucination rates.
Methods
This computational comparative study analyzed 437 multiple-choice questions (1748 queries) from 3 sources: nurse anesthetist school examinations (infirmier anesthésiste diplômé d’État [registered nurse anesthetist]; n=100, 22.9%), European Diploma in Anaesthesiology and Intensive Care (EDAIC; n=219, 50.1%), and EDAIC On-Line Assessment (n=118, 27.0%). Each question was submitted to 4 LLMs (Claude Sonnet 4.5, Gemini 2.5 Pro, GPT-5, and Grok 4) using standardized prompts via default web interface settings. Responses were evaluated through structured consensus review by 2 examiners for accuracy, hallucinations, and question-related confusion. Statistical analysis included Friedman and Wilcoxon signed-rank tests with Holm-Bonferroni correction, the Cochran Q test, and generalized estimating equations.
Results
Average success rates ranged from 86% (SD 18%) to 94% (SD 10%) across LLMs and examination types, exceeding the EDAIC part I passing threshold, representing substantial improvement over previously reported GPT-3.5 performance. For the EDAIC, overall intermodel differences were significant (Friedman χ 2 3 =13.9; P =.003; W =0.02), with Gemini outperforming GPT-5 as the only pairwise difference. Hallucination rates ranged from 11% (11/100) to 20.1% (44/219) without significant intermodel differences. All models exceeded the EDAIC passing threshold.
Conclusions
Current-generation LLMs demonstrated consistently high performance across multiple European anesthesiology examinations but continue to produce clinically relevant hallucinations, supporting their role as supervised educational tools rather than autonomous learning resources. These findings underscore the need for structured integration frameworks and systematic verification when deploying LLMs as learning tools in medical education.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Anesthesiology and Intensive Care, CHU Bichat Claude Bernard, Assistance Publique, Hôpitaux de Paris, Paris, Île-de-France, France.
- Group of Data Modeling, Computational Biology, and Predictive Medicine, Applied Mathematics, Institut de Biologie de l'École Normale Supérieure, Paris, Île-de-France, France.
- Discipline of Anaesthesiology and Intensive Care, Carol Davila University of Medicine and Pharmacy, Sos Fundeni, 258, Bucharest, București, 022322, Romania, 40 742185533.
- 2nd of Department of Anaesthesiology and Intensive Care, Institutul de Urgenţă pentru Boli Cardiovasculare "Prof.Dr. C.C. Iliescu", Bucharest, București, Romania.
- Department of Anaesthesiology and Intensive Care, Bucharest Emergency University Hospital, Bucharest, București, Romania.
- Diabète athérothrombose Thérapies Réunion Océan Indien (DéTROI), INSERM UMR 1188, University of Reunion Island, Saint-Pierre de La Réunion, Réunion, Réunion.
ORCID-Profile: Stefan Andrei, Thibault Giet, Alexis Belouard, Mihai Stefan, Sébastien Tanaka, Philippe Montravers, Aurélie Gouel
Zitieren
APA
Andrei, S., Giet, T., Belouard, A., Stefan, M., Popescu, M., Tanaka, S., Montravers, P., & Gouel, A. (2026). Performance and Hallucination Analysis of Large Language Models on European Anesthesiology Examinations: Cross-Sectional Comparative Study. JMIR Formative Research, 10, e95859-e95859. https://doi.org/10.2196/95859
Vancouver
Andrei S, Giet T, Belouard A, Stefan M, Popescu M, Tanaka S, et al. Performance and Hallucination Analysis of Large Language Models on European Anesthesiology Examinations: Cross-Sectional Comparative Study. JMIR Formative Research. 2026;10:e95859-e95859. doi:10.2196/95859
BibTeX
@article{Andrei2026,
author = {Andrei, Stefan and Giet, Thibault and Belouard, Alexis and Stefan, Mihai and Popescu, Mihai and Tanaka, Sébastien and Montravers, Philippe and Gouel, Aurélie},
title = {Performance and Hallucination Analysis of Large Language Models on European Anesthesiology Examinations: Cross-Sectional Comparative Study},
journal = {JMIR Formative Research},
year = {2026},
volume = {10},
pages = {e95859--e95859},
doi = {10.2196/95859}
}
EUVIMED