EUVIMEDScopri EUVIMED
VergleichsstudieOpen access

AI-Assisted Angoff Standard Setting for Multiple-Choice Examinations in Medical Education: Comparative Study

Salah Eldin Kassab iD, Mariam Shadan iD, Arina Ziganshina iD, Shifan Khanday iD, Hossam Hamdy iD

JMIR Medical Education · 2026-09-09 · 12 · e98766-e98766

DOI: 10.2196/98766 ↗

At a glance

Vergleichsstudie in JMIR Medical Education, published online on 2026-09-09. Written by Salah Eldin Kassab and 4 co-authors. Institution: Department of Biomedical Sciences, College of Medicine, Dubai Medical University, Al Muhaisnah 1, Al Mizhar, Dubai, United Arab Emirates, 971 5058354…. Major MEDLINE topics: Educational Measurement, Artificial Intelligence, Education, Medical. Cited 1 times so far (Crossref); 26 references. The full text is freely available in PubMed Central.

AbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Background

Standard setting is essential for a defensible assessment in medical education. The modified Angoff method requires several expert judges, and determining the minimally competent candidate is cognitively challenging. However, empirical evidence on the role of AI in standard setting is unclear.

Objective

This study aimed to examine the role of large language models (LLMs) in the modified Angoff standard setting method for multiple-choice questions in a basic medical science examination compared with faculty judges.

Methods

This study was conducted in year 2 of the MD program in the United Arab Emirates. Ten faculty judges and 5 LLMs (GPT-5.2, Grok, DeepSeek, MedGemma, and Claude 4.5 Sonnet) determined the Angoff cutoff scores for a summative examination (120 multiple-choice questions). Standardized prompts were used for the LLMs to mimic the same information provided for faculty judges. Generalizability ( G ) theory analyses were performed using a fully crossed item × rater design to estimate variance components, G and Φ coefficients, decision study, and root mean squared error (RMSE) of the Angoff cutoff scores.

Results

Faculty-generated Angoff estimates (mean 69.13, SD 10.92) were comparable to LLM-generated estimates (mean 68.94, SD 12.39). A 2-tailed paired-sample t test revealed no statistically significant difference between the 2 groups (95% CI −2.21 to 2.91; t 119 =0.27; P =.79). Generalizability theory analysis demonstrated moderate reliability for the faculty panel ( G coefficient=0.738; Φ coefficient=0.692). Despite comprising only 5 LLMs, the LLM panel demonstrated higher reliability ( G coefficient=0.823; Φ coefficient=0.815), lower RMSE (1.28 vs 2.33), higher item-related variance (46.85% vs 18.36%), and lower rater-related variance (2.64% vs 16.40%) than the faculty panel. Pass rates were similar using LLM- and faculty-derived cutoff scores (52/73, 71.2%). LLMs differed in their minimally competent candidate conceptualization and approaches to determining item-level percentages of correct responses. Furthermore, the correlations between Angoff estimates and item-related P values were larger in LLMs than in faculty judges ( r =0.552 vs 0.437).

Conclusions

In this single-institution study, the evaluated LLMs generated modified Angoff estimates that were broadly comparable to those of faculty judges. Generalizability theory analyses demonstrated higher G and Φ coefficients, lower rater-related variance, and lower RMSE for the evaluated LLM outputs under standardized prompting conditions, indicating greater consistency in the generated estimates. Application of the resulting cutoff scores produced pass and fail rates similar to those derived from faculty judges. These findings suggest the future application of LLMs as decision assistance tools for modified Angoff standard setting while maintaining expert human oversight.

Subjects

MeSH terms (MEDLINE)

Educational MeasurementArtificial IntelligenceEducation, MedicalHumansLarge Language ModelsUnited Arab Emirates

Author keywords

AIartificial intelligencelarge language modelsmedical educationmodified Angoff methodmultiple-choice examinationsstandard setting

Authors and institutions

  • Department of Biomedical Sciences, College of Medicine, Dubai Medical University, Al Muhaisnah 1, Al Mizhar, Dubai, United Arab Emirates, 971 505835469.
  • Department of Physiology, Faculty of Medicine, Suez Canal University, Ismailia, Egypt.
  • Department of Clinical Sciences, College of Medicine, Dubai Medical University, Dubai, United Arab Emirates.

ORCID profiles: Salah Eldin Kassab, Mariam Shadan, Arina Ziganshina, Shifan Khanday, Hossam Hamdy

Cite

APA

Kassab, S. E., Shadan, M., Ziganshina, A., Khanday, S., & Hamdy, H. (2026). AI-Assisted Angoff Standard Setting for Multiple-Choice Examinations in Medical Education: Comparative Study. JMIR Medical Education, 12, e98766-e98766. https://doi.org/10.2196/98766

Vancouver

Kassab SE, Shadan M, Ziganshina A, Khanday S, Hamdy H. AI-Assisted Angoff Standard Setting for Multiple-Choice Examinations in Medical Education: Comparative Study. JMIR Medical Education. 2026;12:e98766-e98766. doi:10.2196/98766

BibTeX
@article{Kassab2026,
  author = {Kassab, Salah Eldin and Shadan, Mariam and Ziganshina, Arina and Khanday, Shifan and Hamdy, Hossam},
  title = {AI-Assisted Angoff Standard Setting for Multiple-Choice Examinations in Medical Education: Comparative Study},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e98766--e98766},
  doi = {10.2196/98766}
}
Start for free →