EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

A Comparative Analysis of Large Language Model Performance on USMLE Step 1-Style Allergy/Immunology Questions: Evaluating Correctness and Consistency

Moshe Carroll iD, Sabrina Kentis, Hannah Kareff, Clyde Schechter, Sunit Jariwala iD

Applied Clinical Informatics · 2026-08 · 17 (04) · 754-762

DOI: 10.1055/a-2946-7393 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Applied Clinical Informatics, online erschienen am 07.09.2026. Verfasst von Moshe Carroll und 4 weiteren Autor:innen. Beteiligte Einrichtung: Albert Einstein College of Medicine, Montefiore Medical Center, Department of Medicine, New York, United States, Bronx. Hauptthemen laut MEDLINE: Large Language Models, Allergy and Immunology, Educational Measurement, Licensure, Medical. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) are rapidly transforming medical education, yet their performance in Allergy/Immunology remains insufficiently characterized. Furthermore, concerns regarding accuracy, consistency, and sensitivity to input format persist.

Objectives

This study aimed to evaluate and compare the accuracy and response consistency of three leading LLMs—ChatGPT-5, Gemini 2.5, and Grok 4—on Allergy/Immunology United States Medical Licensing Examination (USMLE) Step 1-style questions under different prompt conditions.

Methods

Thirty-five USMLE Step 1-style questions were selected. Questions were presented to each model in two formats: single-question prompts and a combined prompt containing all questions. Fifteen trials were conducted for each format per model. Performance was assessed using mean accuracy, and variability was measured using Shannon entropy. Mixed-effects models tested the effects of model, prompt condition, and question difficulty.

Results

Overall accuracy differed significantly (p < 0.001), with Gemini (80.7%) and Grok (80.5%) achieving higher mean scores than ChatGPT (74.3%). Single-item prompts yielded superior performance, with Grok (93.1%) and Gemini (90.9%) demonstrating the highest accuracy. Transitioning to a combined prompt significantly reduced accuracy for all models. Accuracy also decreased with increasing question difficulty for all models. Grok demonstrated superior reliability, maintaining the lowest overall response entropy, whereas ChatGPT exhibited the highest variability.

Conclusion

On Allergy/Immunology Step 1-style questions, Gemini and Grok demonstrated higher accuracy than ChatGPT, although their overall accuracies remained approximately 81%. Grok offered the most consistent performance. All models demonstrated substantial sensitivity to prompt complexity and inherent performance limitations. These findings underscore the importance of prompt optimization and support the supplementary role of these models in medical education.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsAllergy and ImmunologyEducational MeasurementLicensure, MedicalUnited StatesHumans

Autor:innen und Institutionen

  • Albert Einstein College of Medicine, Montefiore Medical Center, Department of Medicine, New York, United States, Bronx.
  • Albert Einstein College of Medicine, Department of Family and Social Medicine, New York, United States, Bronx.

ORCID-Profile: Moshe Carroll, Sunit Jariwala

Zitieren

APA

Carroll, M., Kentis, S., Kareff, H., Schechter, C., & Jariwala, S. (2026). A Comparative Analysis of Large Language Model Performance on USMLE Step 1-Style Allergy/Immunology Questions: Evaluating Correctness and Consistency. Applied Clinical Informatics, 17(04), 754-762. https://doi.org/10.1055/a-2946-7393

Vancouver

Carroll M, Kentis S, Kareff H, Schechter C, Jariwala S. A Comparative Analysis of Large Language Model Performance on USMLE Step 1-Style Allergy/Immunology Questions: Evaluating Correctness and Consistency. Applied Clinical Informatics. 2026;17(04):754-762. doi:10.1055/a-2946-7393

BibTeX
@article{Carroll2026,
  author = {Carroll, Moshe and Kentis, Sabrina and Kareff, Hannah and Schechter, Clyde and Jariwala, Sunit},
  title = {A Comparative Analysis of Large Language Model Performance on USMLE Step 1-Style Allergy/Immunology Questions: Evaluating Correctness and Consistency},
  journal = {Applied Clinical Informatics},
  year = {2026},
  volume = {17},
  number = {04},
  pages = {754--762},
  doi = {10.1055/a-2946-7393}
}
Kostenlos starten →