EUVIMEDEUVIMED entdecken
Open Access

Evaluating large language model performance to support the diagnosis and management of patients with primary immune disorders

Nicholas L Rider, Yingya Li, Aaron T Chin, Daniel V DiGiacomo, Cullen Dutmer, Jocelyn R Farmer, Kirk Roberts, Guergana Savova, Mei-Sing Ong

Journal of Allergy and Clinical Immunology · 2025

DOI: 10.1016/j.jaci.2025.02.004 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Allergy and Clinical Immunology, 2025. Verfasst von Nicholas L Rider und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Health Systems and Implementation Science, Virginia Tech Carilion School of Medicine, Roanoke, Va; Section of Allergy and Immunology, D…. Hauptthemen laut MEDLINE: Artificial Intelligence, Immune System Diseases. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIGMS NIH HHS, NIMHD NIH HHS, NIAID NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

BACKGROUND: Generative artificial intelligence (GAI) is transforming health care in a variety of ways; however, the present utility of GAI for supporting clinicians who treat rare disease such as primary immune disorders (PIs) is not well studied. We evaluated the ability of 6 state-of-the-art large language models (LLMs) for providing clinical guidance about PIs. OBJECTIVE: To quantitatively and qualitatively measure the utility of current, open-source LLMs for diagnosing and providing helpful clinical decision support about PIs. METHODS: Five expert clinical immunologists each provided 5 real-world, anonymized PI case vignettes via multi-turn prompting to 6 LLMs (OpenAI GPT-4o, Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct, Mistral-7B-Instruct-v0.3, Mistral-Large-Instruct-2407, Mixtral-8x7B-Instruct-v0.1). We assessed the diagnostic accuracy of the LLMs and the quality of clinical reasoning using the Revised-IDEA (R-IDEA) score. Qualitative LLM assessment was made by immunologist narratives. RESULTS: Performance accuracy (>88%) and R-IDEA scores (≥8) were superior for 3 models (GPT-4o, Llama-3.1-70B-Instruct, Mistral-Large-Instruct-2407), with GPT-4o achieving the highest diagnostic accuracy (96.2%). Conversely, the remaining 3 models fell below acceptable accuracy rates near 60% or lower and had poor R-IDEA scores (≤0.55), with Mistral-7B-Instruct-v0.3 attaining the worst diagnostic accuracy (42.3%). Compared with the 3 best-performing LLMs, the 3 worst-performing LLMs had a substantially lower median R-IDEA score (P < .001). Interclass correlation coefficient for R-IDEA score assignments varied substantially by LLM, ranging from good to poor agreement, and did not appear to correlate with either diagnostic accuracy or median R-IDEA score. Qualitatively, immunologists identified several themes (eg, correctness, differential diagnosis appropriateness, relative conciseness of explanations) of relevance to PIs. CONCLUSIONS: LLM can support diagnosis and management of PIs; however, further tuning is needed to optimize LLMs for best practice recommendations.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceImmune System DiseasesHumansMaleLanguageFemaleLarge Language Models

Stichwörter der Autor:innen

Primary immune disordersgenerative AIhealth care AIinborn errors of immunitylarge language modelsmedical chatbot

Autor:innen und Institutionen

  • Department of Health Systems and Implementation Science, Virginia Tech Carilion School of Medicine, Roanoke, Va; Section of Allergy and Immunology, Department of Medicine, Carilion Clinic, Roanoke, Va. Electronic addres…
  • Computational Health Informatics Program, Harvard Medical School and Boston Children's Hospital, Boston, Mass.
  • Department of Pediatrics, Division of Immunology, Allergy, and Rheumatology, University of California, Los Angeles, Los Angeles, Calif.
  • Department of Pediatrics, K. Hovnanian Children's Hospital, Jersey Shore University Medical Center, Neptune, NJ; Hackensack Meridian School of Medicine, Nutley, NJ.
  • Department of Pediatrics, University of Colorado School of Medicine, Denver, Colo; Section of Allergy & Immunology, Children's Hospital Colorado, Denver, Colo.
  • Clinical Immunodeficiency Program of Beth Israel Lahey Health, Division of Allergy and Immunology, Lahey Hospital & Medical Center, Burlington, Mass; Department of Medicine, UMass Chan Medical School, Burlington, Mass.

Förderung

  • NIGMS NIH HHS
  • NIMHD NIH HHS
  • NIAID NIH HHS
  • Centers for Disease Control and Prevention
  • Patient-Centered Outcomes Research Institute
  • National Institute of Allergy and Infectious Diseases
  • National Institute on Minority Health and Health Disparities
  • National Institutes of Health

Zitieren

APA

Rider, N. L., Li, Y., Chin, A. T., DiGiacomo, D. V., Dutmer, C., Farmer, J. R., Roberts, K., Savova, G., & Ong, M. S. (2025). Evaluating large language model performance to support the diagnosis and management of patients with primary immune disorders. Journal of Allergy and Clinical Immunology. https://doi.org/10.1016/j.jaci.2025.02.004

Vancouver

Rider NL, Li Y, Chin AT, DiGiacomo DV, Dutmer C, Farmer JR, et al. Evaluating large language model performance to support the diagnosis and management of patients with primary immune disorders. Journal of Allergy and Clinical Immunology. 2025. doi:10.1016/j.jaci.2025.02.004

BibTeX
@article{Rider2025,
  author = {Rider, Nicholas L and Li, Yingya and Chin, Aaron T and DiGiacomo, Daniel V and Dutmer, Cullen and Farmer, Jocelyn R and Roberts, Kirk and Savova, Guergana and Ong, Mei-Sing},
  title = {Evaluating large language model performance to support the diagnosis and management of patients with primary immune disorders},
  journal = {Journal of Allergy and Clinical Immunology},
  year = {2025},
  doi = {10.1016/j.jaci.2025.02.004}
}
Kostenlos starten →