A Comparative Analysis of Large Language Model Performance on USMLE Step 1-Style Allergy/Immunology Questions: Evaluating Correctness and Consistency
Applied Clinical Informatics · 2026-08 · 17 (04) · 754-762
DOI: 10.1055/a-2946-7393 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Applied Clinical Informatics, online erschienen am 07.09.2026. Verfasst von Moshe Carroll und 4 weiteren Autor:innen. Beteiligte Einrichtung: Albert Einstein College of Medicine, Montefiore Medical Center, Department of Medicine, New York, United States, Bronx. Hauptthemen laut MEDLINE: Large Language Models, Allergy and Immunology, Educational Measurement, Licensure, Medical. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models (LLMs) are rapidly transforming medical education, yet their performance in Allergy/Immunology remains insufficiently characterized. Furthermore, concerns regarding accuracy, consistency, and sensitivity to input format persist.
Objectives
This study aimed to evaluate and compare the accuracy and response consistency of three leading LLMs—ChatGPT-5, Gemini 2.5, and Grok 4—on Allergy/Immunology United States Medical Licensing Examination (USMLE) Step 1-style questions under different prompt conditions.
Methods
Thirty-five USMLE Step 1-style questions were selected. Questions were presented to each model in two formats: single-question prompts and a combined prompt containing all questions. Fifteen trials were conducted for each format per model. Performance was assessed using mean accuracy, and variability was measured using Shannon entropy. Mixed-effects models tested the effects of model, prompt condition, and question difficulty.
Results
Overall accuracy differed significantly (p < 0.001), with Gemini (80.7%) and Grok (80.5%) achieving higher mean scores than ChatGPT (74.3%). Single-item prompts yielded superior performance, with Grok (93.1%) and Gemini (90.9%) demonstrating the highest accuracy. Transitioning to a combined prompt significantly reduced accuracy for all models. Accuracy also decreased with increasing question difficulty for all models. Grok demonstrated superior reliability, maintaining the lowest overall response entropy, whereas ChatGPT exhibited the highest variability.
Conclusion
On Allergy/Immunology Step 1-style questions, Gemini and Grok demonstrated higher accuracy than ChatGPT, although their overall accuracies remained approximately 81%. Grok offered the most consistent performance. All models demonstrated substantial sensitivity to prompt complexity and inherent performance limitations. These findings underscore the importance of prompt optimization and support the supplementary role of these models in medical education.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Albert Einstein College of Medicine, Montefiore Medical Center, Department of Medicine, New York, United States, Bronx.
- Albert Einstein College of Medicine, Department of Family and Social Medicine, New York, United States, Bronx.
ORCID-Profile: Moshe Carroll, Sunit Jariwala
Zitieren
APA
Carroll, M., Kentis, S., Kareff, H., Schechter, C., & Jariwala, S. (2026). A Comparative Analysis of Large Language Model Performance on USMLE Step 1-Style Allergy/Immunology Questions: Evaluating Correctness and Consistency. Applied Clinical Informatics, 17(04), 754-762. https://doi.org/10.1055/a-2946-7393
Vancouver
Carroll M, Kentis S, Kareff H, Schechter C, Jariwala S. A Comparative Analysis of Large Language Model Performance on USMLE Step 1-Style Allergy/Immunology Questions: Evaluating Correctness and Consistency. Applied Clinical Informatics. 2026;17(04):754-762. doi:10.1055/a-2946-7393
BibTeX
@article{Carroll2026,
author = {Carroll, Moshe and Kentis, Sabrina and Kareff, Hannah and Schechter, Clyde and Jariwala, Sunit},
title = {A Comparative Analysis of Large Language Model Performance on USMLE Step 1-Style Allergy/Immunology Questions: Evaluating Correctness and Consistency},
journal = {Applied Clinical Informatics},
year = {2026},
volume = {17},
number = {04},
pages = {754--762},
doi = {10.1055/a-2946-7393}
}
EUVIMED