EUVIMEDObjevte EUVIMED
QuerschnittstudieOpen access

Ambiguity Detection in Medical Exams via Large Language Models: Retrospective Cross-Sectional Pilot Study

Romain Lombardi iD, Alexandre Destere iD, Jean Dellamonica iD, Alexandre O Gérard iD, Mathieu Jozwiak iD

JMIR Medical Education · 2026-05-26 · 12 · e82702-e82702

DOI: 10.2196/82702 ↗

At a glance

Querschnittstudie in JMIR Medical Education, published online on 2026-05-26. Written by Romain Lombardi and 4 co-authors. Institution: Critical Care Unit, Pasteur 2 University Hospital, 30 Voie Romaine, Nice, 06100, France, 33 0669032616. Major MEDLINE topics: Large Language Models, Educational Measurement. 34 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Large language models (LLMs) have emerged as promising tools in medical education due to their ability to understand, generate, and reason with natural language. Their ability to simulate expert reasoning suggests a potential for supporting quality control in assessment design. In this study, the use of LLMs in identifying ambiguous or poorly constructed exam items in critical care academic assessments was evaluated.

Objective

The study aimed to develop automated ambiguity and quality scores to objectively assess individual questions and entire exam components.

Methods

We analyzed 264 questions from academic exams conducted over 3 academic years (2023-2025) at the Medical School of Université Côte d’Azur. Questions were drawn from 4 docimological formats: progressive clinical cases (PCC), mini-PCC, key feature problems, and isolated question sequences (IQS). Each element was submitted to 4 LLMs (ChatGPT, Gemini Pro, Le Chat, and DeepSeek) without prompt engineering. Performance was evaluated using the official correction key. We applied 4 binary diagnostic tags based on model agreement and self-reported ambiguity: ambiguity, low performance, incoherence, and subjective ambiguity. These tags generated a composite ambiguity score and contributed to a weighted quality score for each exam component.

Results

LLMs achieved mean scores in the same range as students, with no significant differences across academic years and significantly higher performance on the mini-PCC and IQS formats ( P =.049 and P =.04, respectively). IQS items had the highest ambiguity scores (54 items received a score of 2 in both 2023 and 2024, and 53 items retained the same score). Tag patterns revealed frequent issues with ambiguity and inconsistency. Quality scores varied across academic years. IQS predominantly showed moderate ambiguity (score 2), with occasional instances of strong signals. There was no significant difference in quality based on author specialty or seniority ( P =.08 and P =.44, respectively).

Conclusions

In this pilot study, LLMs may offer a preliminary framework to proactively detect ambiguous exam questions and estimate the overall quality of an exam. Integrating these tools into the assessment design process could potentially reduce the need for postexam corrections and may help improve fairness and clarity in medical evaluations.

Subjects

MeSH terms (MEDLINE)

Large Language ModelsEducational MeasurementPilot ProjectsHumansRetrospective StudiesCross-Sectional Studies

Author keywords

LLMambiguity detectionartificial intelligenceautomated scoringcritical caredocimologyemergencyexamslarge language modelmedical educationquality assessment

Authors and institutions

  • Critical Care Unit, Pasteur 2 University Hospital, 30 Voie Romaine, Nice, 06100, France, 33 0669032616.
  • UR2CA, Unité de Recherche Clinique Côte d'Azur, Université Côte d'Azur, Nice, France.
  • Department of Clinical Pharmacology and Pharmacovigilance Center, Medical Centre, Université Côte d'Azur, Nice, France.
  • Maasai Team, Laboratoire J.A. Dieudonné, Université Côte d'Azur, Inria, CNRS, Nice, France.
  • Critical Care Unit, Archet 1 University Hospital, Nice, France.

ORCID profiles: Romain Lombardi, Alexandre Destere, Jean Dellamonica, Alexandre O Gérard, Mathieu Jozwiak

Cite

APA

Lombardi, R., Destere, A., Dellamonica, J., Gérard, A. O., & Jozwiak, M. (2026). Ambiguity Detection in Medical Exams via Large Language Models: Retrospective Cross-Sectional Pilot Study. JMIR Medical Education, 12, e82702-e82702. https://doi.org/10.2196/82702

Vancouver

Lombardi R, Destere A, Dellamonica J, Gérard AO, Jozwiak M. Ambiguity Detection in Medical Exams via Large Language Models: Retrospective Cross-Sectional Pilot Study. JMIR Medical Education. 2026;12:e82702-e82702. doi:10.2196/82702

BibTeX
@article{Lombardi2026,
  author = {Lombardi, Romain and Destere, Alexandre and Dellamonica, Jean and Gérard, Alexandre O and Jozwiak, Mathieu},
  title = {Ambiguity Detection in Medical Exams via Large Language Models: Retrospective Cross-Sectional Pilot Study},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e82702--e82702},
  doi = {10.2196/82702}
}
Start for free →