EUVIMEDDescoperă EUVIMED
VergleichsstudieQuerschnittstudieOpen access

Performance and Hallucination Analysis of Large Language Models on European Anesthesiology Examinations: Cross-Sectional Comparative Study

Stefan Andrei iD, Thibault Giet iD, Alexis Belouard iD, Mihai Stefan iD, Mihai Popescu, Sébastien Tanaka iD, Philippe Montravers iD, Aurélie Gouel iD

JMIR Formative Research · 2026-09-01 · 10 · e95859-e95859

DOI: 10.2196/95859 ↗

At a glance

Vergleichsstudie in JMIR Formative Research, published online on 2026-09-01. Written by Stefan Andrei and 7 co-authors. Institution: Department of Anesthesiology and Intensive Care, CHU Bichat Claude Bernard, Assistance Publique, Hôpitaux de Paris, Paris, Île-de-France, France. Major MEDLINE topics: Large Language Models, Anesthesiology, Educational Measurement, Hallucinations. 30 references. The full text is freely available in PubMed Central.

RezumatSubjectsDate bibliograficeCiteRelated studies

Abstract (original language)

Background

Large language models (LLMs) have shown promising performance on medical examinations across specialties. However, comparative evaluations of current-generation LLMs across multiple European anesthesiology examinations, alongside structured assessment of hallucinations vs question-related confusion, remain lacking.

Objective

This study aimed to compare the performance of 4 state-of-the-art LLMs on anesthesiology and intensive medicine examination questions and assess their hallucination rates.

Methods

This computational comparative study analyzed 437 multiple-choice questions (1748 queries) from 3 sources: nurse anesthetist school examinations (infirmier anesthésiste diplômé d’État [registered nurse anesthetist]; n=100, 22.9%), European Diploma in Anaesthesiology and Intensive Care (EDAIC; n=219, 50.1%), and EDAIC On-Line Assessment (n=118, 27.0%). Each question was submitted to 4 LLMs (Claude Sonnet 4.5, Gemini 2.5 Pro, GPT-5, and Grok 4) using standardized prompts via default web interface settings. Responses were evaluated through structured consensus review by 2 examiners for accuracy, hallucinations, and question-related confusion. Statistical analysis included Friedman and Wilcoxon signed-rank tests with Holm-Bonferroni correction, the Cochran Q test, and generalized estimating equations.

Results

Average success rates ranged from 86% (SD 18%) to 94% (SD 10%) across LLMs and examination types, exceeding the EDAIC part I passing threshold, representing substantial improvement over previously reported GPT-3.5 performance. For the EDAIC, overall intermodel differences were significant (Friedman χ 2 3 =13.9; P =.003; W =0.02), with Gemini outperforming GPT-5 as the only pairwise difference. Hallucination rates ranged from 11% (11/100) to 20.1% (44/219) without significant intermodel differences. All models exceeded the EDAIC passing threshold.

Conclusions

Current-generation LLMs demonstrated consistently high performance across multiple European anesthesiology examinations but continue to produce clinically relevant hallucinations, supporting their role as supervised educational tools rather than autonomous learning resources. These findings underscore the need for structured integration frameworks and systematic verification when deploying LLMs as learning tools in medical education.

Subjects

MeSH terms (MEDLINE)

Large Language ModelsAnesthesiologyEducational MeasurementHallucinationsHumansCross-Sectional StudiesEurope

Author keywords

AIanesthesiologyartificial intelligenceexaminationhallucinationsintensive care medicinelarge language modelsmedical educationmultiple-choice questions

Authors and institutions

  • Department of Anesthesiology and Intensive Care, CHU Bichat Claude Bernard, Assistance Publique, Hôpitaux de Paris, Paris, Île-de-France, France.
  • Group of Data Modeling, Computational Biology, and Predictive Medicine, Applied Mathematics, Institut de Biologie de l'École Normale Supérieure, Paris, Île-de-France, France.
  • Discipline of Anaesthesiology and Intensive Care, Carol Davila University of Medicine and Pharmacy, Sos Fundeni, 258, Bucharest, București, 022322, Romania, 40 742185533.
  • 2nd of Department of Anaesthesiology and Intensive Care, Institutul de Urgenţă pentru Boli Cardiovasculare "Prof.Dr. C.C. Iliescu", Bucharest, București, Romania.
  • Department of Anaesthesiology and Intensive Care, Bucharest Emergency University Hospital, Bucharest, București, Romania.
  • Diabète athérothrombose Thérapies Réunion Océan Indien (DéTROI), INSERM UMR 1188, University of Reunion Island, Saint-Pierre de La Réunion, Réunion, Réunion.

ORCID profiles: Stefan Andrei, Thibault Giet, Alexis Belouard, Mihai Stefan, Sébastien Tanaka, Philippe Montravers, Aurélie Gouel

Cite

APA

Andrei, S., Giet, T., Belouard, A., Stefan, M., Popescu, M., Tanaka, S., Montravers, P., & Gouel, A. (2026). Performance and Hallucination Analysis of Large Language Models on European Anesthesiology Examinations: Cross-Sectional Comparative Study. JMIR Formative Research, 10, e95859-e95859. https://doi.org/10.2196/95859

Vancouver

Andrei S, Giet T, Belouard A, Stefan M, Popescu M, Tanaka S, et al. Performance and Hallucination Analysis of Large Language Models on European Anesthesiology Examinations: Cross-Sectional Comparative Study. JMIR Formative Research. 2026;10:e95859-e95859. doi:10.2196/95859

BibTeX
@article{Andrei2026,
  author = {Andrei, Stefan and Giet, Thibault and Belouard, Alexis and Stefan, Mihai and Popescu, Mihai and Tanaka, Sébastien and Montravers, Philippe and Gouel, Aurélie},
  title = {Performance and Hallucination Analysis of Large Language Models on European Anesthesiology Examinations: Cross-Sectional Comparative Study},
  journal = {JMIR Formative Research},
  year = {2026},
  volume = {10},
  pages = {e95859--e95859},
  doi = {10.2196/95859}
}
Start for free →