EUVIMEDOdkrijte EUVIMED
Crossref

Use of Large Language Models for Automated Short-Answer Question Grading in Second-Cycle Medical Education: Performance and Consistency Evaluation Study

Sabrina Sidali iD, Pierre-Antoine Soret iD, Mathilde Wagner iD, Thibault Voron iD, Quentin Vanderbecq iD

JMIR Medical Education · 2026-11-02 · 12 · e89445

DOI: 10.2196/89445 ↗

At a glance

Journal article in JMIR Medical Education, published online on 2026-11-02. Written by Sabrina Sidali and 4 co-authors. Institution: Cordeliers Research Center, Sorbonne University, Inserm, University of Paris, Functional Genomics of Solid Tumors Team, French National Cancer League…. Major MEDLINE topics: Large Language Models, Educational Measurement, Education, Medical. 21 references.

PovzetekSubjectsBibliografski podatkiCiteRelated studies

Abstract (original language)

Background

Short-answer questions (SAQs) effectively assess higher-order reasoning skills in medical education but face challenges in scalability due to time-intensive manual grading and concerns about interrater reliability. Recent advances in large language models (LLMs) may offer a solution through automated assessment.

Objective

This study evaluated the performance and consistency of LLMs—ChatGPT, Claude Sonnet 4, and Mistral Pro—compared with those of human graders for SAQ grading in hepatology and gastroenterology.

Methods

We retrospectively analyzed SAQs from hepatology and gastroenterology examinations at Sorbonne University, Paris, France (2023-2025). Three human evaluators (2 hepatogastroenterologists and 1 radiologist) independently graded all responses twice, with the majority consensus score serving as the reference standard. Six LLMs were evaluated: ChatGPT (GPT-4o, GPT-4.5, GPT-4o3, and GPT-5), Claude Sonnet 4, and Mistral Pro. Each LLM graded all responses twice using standardized prompts, including questions, model answers, grading guidelines, and scoring rubrics. Agreement was assessed using quadratic weighted kappa (QWK), Cohen κ, the intraclass correlation coefficient (ICC), and the mean absolute error (MAE) for interhuman, intrahuman, LLM-versus-human, and intramodel comparisons.

Results

A total of 902 student responses from 12 SAQs were included and analyzed. Interhuman grader agreement was almost perfect (QWK 0.90-0.92; ICC 0.90-0.92; MAE 0.05-0.07). Claude Sonnet 4 and GPT-4o3 demonstrated near-human–level performance when compared with the majority vote of human evaluators (Claude Sonnet 4: QWK 0.88-0.90; ICC 0.88-0.90; MAE 0.06-0.07 and GPT-4o3: QWK 0.82-0.89; ICC 0.82-0.89; MAE 0.07-0.09). GPT-5 showed substantial agreement (QWK 0.77-0.79). Mistral Pro and older ChatGPT versions performed poorly (QWK

Subjects

MeSH terms (MEDLINE)

Large Language ModelsEducational MeasurementEducation, MedicalHumansRetrospective StudiesReproducibility of Results

Author keywords

AIartificial intelligencelarge language modelsmedical educationshort-answer grading

Authors and institutions

  • Cordeliers Research Center, Sorbonne University, Inserm, University of Paris, Functional Genomics of Solid Tumors Team, French National Cancer League-Accredited Team, Labex OncoImmunology, F-75006 Paris, Île-de-France,…
  • University of Paris, Greater Paris University Hospitals (AP-HP), DMU DIGEST, Beaujon Hospital, Reference Center for Vascular Liver Diseases, FILFOIE, ERN RARE-LIVER, Inflammation Research Center, Inserm UMR 1149, Paris,…
  • Sorbonne University, Inserm, Saint-Antoine Research Center (CRSA), Paris, Île-de-France, France.
  • Saint-Antoine Hospital, Greater Paris University Hospitals (AP-HP), Paris, Île-de-France, France.
  • Department of Radiology, Saint-Antoine Hospital, Greater Paris University Hospitals (AP-HP), Paris, Île-de-France, France.
  • LIB, UMR 7371, Sorbonne University, CNRS, Inserm U1146, Paris, Île-de-France, France.

ORCID profiles: Sabrina Sidali, Pierre-Antoine Soret, Mathilde Wagner, Thibault Voron, Quentin Vanderbecq

Cite

APA

Sidali, S., Soret, P. A., Wagner, M., Voron, T., & Vanderbecq, Q. (2026). Use of Large Language Models for Automated Short-Answer Question Grading in Second-Cycle Medical Education: Performance and Consistency Evaluation Study. JMIR Medical Education, 12, e89445. https://doi.org/10.2196/89445

Vancouver

Sidali S, Soret PA, Wagner M, Voron T, Vanderbecq Q. Use of Large Language Models for Automated Short-Answer Question Grading in Second-Cycle Medical Education: Performance and Consistency Evaluation Study. JMIR Medical Education. 2026;12:e89445. doi:10.2196/89445

BibTeX
@article{Sidali2026,
  author = {Sidali, Sabrina and Soret, Pierre-Antoine and Wagner, Mathilde and Voron, Thibault and Vanderbecq, Quentin},
  title = {Use of Large Language Models for Automated Short-Answer Question Grading in Second-Cycle Medical Education: Performance and Consistency Evaluation Study},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e89445},
  doi = {10.2196/89445}
}
Start for free →