EUVIMEDEUVIMED entdecken
QuerschnittstudieOpen Access

Ambiguity Detection in Medical Exams via Large Language Models: Retrospective Cross-Sectional Pilot Study

Romain Lombardi iD, Alexandre Destere iD, Jean Dellamonica iD, Alexandre O Gérard iD, Mathieu Jozwiak iD

JMIR Medical Education · 26.05.2026 · 12 · e82702-e82702

DOI: 10.2196/82702 ↗

Kurz gefasst

Querschnittstudie in der Zeitschrift JMIR Medical Education, online erschienen am 26.05.2026. Verfasst von Romain Lombardi und 4 weiteren Autor:innen. Beteiligte Einrichtung: Critical Care Unit, Pasteur 2 University Hospital, 30 Voie Romaine, Nice, 06100, France, 33 0669032616. Hauptthemen laut MEDLINE: Large Language Models, Educational Measurement. Die Arbeit nennt 34 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) have emerged as promising tools in medical education due to their ability to understand, generate, and reason with natural language. Their ability to simulate expert reasoning suggests a potential for supporting quality control in assessment design. In this study, the use of LLMs in identifying ambiguous or poorly constructed exam items in critical care academic assessments was evaluated.

Objective

The study aimed to develop automated ambiguity and quality scores to objectively assess individual questions and entire exam components.

Methods

We analyzed 264 questions from academic exams conducted over 3 academic years (2023-2025) at the Medical School of Université Côte d’Azur. Questions were drawn from 4 docimological formats: progressive clinical cases (PCC), mini-PCC, key feature problems, and isolated question sequences (IQS). Each element was submitted to 4 LLMs (ChatGPT, Gemini Pro, Le Chat, and DeepSeek) without prompt engineering. Performance was evaluated using the official correction key. We applied 4 binary diagnostic tags based on model agreement and self-reported ambiguity: ambiguity, low performance, incoherence, and subjective ambiguity. These tags generated a composite ambiguity score and contributed to a weighted quality score for each exam component.

Results

LLMs achieved mean scores in the same range as students, with no significant differences across academic years and significantly higher performance on the mini-PCC and IQS formats ( P =.049 and P =.04, respectively). IQS items had the highest ambiguity scores (54 items received a score of 2 in both 2023 and 2024, and 53 items retained the same score). Tag patterns revealed frequent issues with ambiguity and inconsistency. Quality scores varied across academic years. IQS predominantly showed moderate ambiguity (score 2), with occasional instances of strong signals. There was no significant difference in quality based on author specialty or seniority ( P =.08 and P =.44, respectively).

Conclusions

In this pilot study, LLMs may offer a preliminary framework to proactively detect ambiguous exam questions and estimate the overall quality of an exam. Integrating these tools into the assessment design process could potentially reduce the need for postexam corrections and may help improve fairness and clarity in medical evaluations.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsEducational MeasurementPilot ProjectsHumansRetrospective StudiesCross-Sectional Studies

Stichwörter der Autor:innen

LLMambiguity detectionartificial intelligenceautomated scoringcritical caredocimologyemergencyexamslarge language modelmedical educationquality assessment

Autor:innen und Institutionen

  • Critical Care Unit, Pasteur 2 University Hospital, 30 Voie Romaine, Nice, 06100, France, 33 0669032616.
  • UR2CA, Unité de Recherche Clinique Côte d'Azur, Université Côte d'Azur, Nice, France.
  • Department of Clinical Pharmacology and Pharmacovigilance Center, Medical Centre, Université Côte d'Azur, Nice, France.
  • Maasai Team, Laboratoire J.A. Dieudonné, Université Côte d'Azur, Inria, CNRS, Nice, France.
  • Critical Care Unit, Archet 1 University Hospital, Nice, France.

ORCID-Profile: Romain Lombardi, Alexandre Destere, Jean Dellamonica, Alexandre O Gérard, Mathieu Jozwiak

Zitieren

APA

Lombardi, R., Destere, A., Dellamonica, J., Gérard, A. O., & Jozwiak, M. (2026). Ambiguity Detection in Medical Exams via Large Language Models: Retrospective Cross-Sectional Pilot Study. JMIR Medical Education, 12, e82702-e82702. https://doi.org/10.2196/82702

Vancouver

Lombardi R, Destere A, Dellamonica J, Gérard AO, Jozwiak M. Ambiguity Detection in Medical Exams via Large Language Models: Retrospective Cross-Sectional Pilot Study. JMIR Medical Education. 2026;12:e82702-e82702. doi:10.2196/82702

BibTeX
@article{Lombardi2026,
  author = {Lombardi, Romain and Destere, Alexandre and Dellamonica, Jean and Gérard, Alexandre O and Jozwiak, Mathieu},
  title = {Ambiguity Detection in Medical Exams via Large Language Models: Retrospective Cross-Sectional Pilot Study},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e82702--e82702},
  doi = {10.2196/82702}
}
Kostenlos starten →