Evaluating large language models for WAO/EAACI guideline compliance in hereditary angioedema management
Allergologia et Immunopathologia · 2025
DOI: 10.15586/aei.v53i4.1353 ↗Kurz gefasst
Leitlinie in der Zeitschrift Allergologia et Immunopathologia, 2025. Verfasst von Mehmet Emin Gerek und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Internal Medicine, Division of Clinical Immunology and Allergy, Necmettin Erbakan University, Faculty of Medicine, Konya, Türkiye; drme…. Hauptthemen laut MEDLINE: Angioedemas, Hereditary, Practice Guidelines as Topic, Guideline Adherence. Die Arbeit wurde bisher 2-mal zitiert (Crossref). Gefördert von: None..
Abstract (Originalsprache)
Introduction
Hereditary angioedema (HAE) is a rare but potentially life-threatening disorder characterized by recurrent swelling episodes. Adherence to clinical guidelines, such as the World Allergy Organization/European Academy of Allergy & Clinical Immunology (WAO/EAACI) guidelines, is crucial for effective management. With the increasing role of artificial intelligence in medicine, large language models (LLMs) offer potential for clinical decision support. This study evaluates the performance of ChatGPT, Gemini, Perplexity, and Copilot in providing guideline-adherent responses for HAE management.
Methods
Twenty-eight key recommendations from the WAO/EAACI HAE guidelines were reformulated into interrogative formats and posed to the selected LLMs. Two independent clinicians assessed responses based on accuracy, adequacy, clarity, and citation reliability using a five-point Likert scale. References were categorized as guideline-based, trustworthy, or untrustworthy. A reevaluation with explicit citation instructions was conducted, with discrepancies resolved by a third reviewer.
Results
ChatGPT and Gemini outperformed Perplexity and Copilot, achieving median accuracy and adequacy scores of 5.0 versus 3.0, respectively. ChatGPT had the lowest rate of unreliable references, whereas Gemini showed inconsistency in citation behavior. Significant differences in response quality were observed among models (p < 0.001). Providing explicit sourcing instructions improved performance consistency, particularly for Gemini.
Conclusion
ChatGPT and Gemini demonstrated superior adherence to WAO/EAACI guidelines, suggesting that LLMs can support clinical decision-making in rare diseases. However, inconsistencies in citation practices highlight the need for further validation and optimization to enhance reliability in medical applications.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Internal Medicine, Division of Clinical Immunology and Allergy, Necmettin Erbakan University, Faculty of Medicine, Konya, Türkiye; drmegerek@gmail.com.
- Department of Internal Medicine, Division of Clinical Immunology and Allergy, Necmettin Erbakan University, Faculty of Medicine, Konya, Türkiye.
Förderung
- None.
Zitieren
APA
Gerek, M. E., Önalan, T., Çölkesen, F., & Arslan, Ş. (2025). Evaluating large language models for WAO/EAACI guideline compliance in hereditary angioedema management. Allergologia et Immunopathologia. https://doi.org/10.15586/aei.v53i4.1353
Vancouver
Gerek ME, Önalan T, Çölkesen F, Arslan Ş. Evaluating large language models for WAO/EAACI guideline compliance in hereditary angioedema management. Allergologia et Immunopathologia. 2025. doi:10.15586/aei.v53i4.1353
BibTeX
@article{Gerek2025,
author = {Gerek, Mehmet Emin and Önalan, Tuğba and Çölkesen, Fatih and Arslan, Şevket},
title = {Evaluating large language models for WAO/EAACI guideline compliance in hereditary angioedema management},
journal = {Allergologia et Immunopathologia},
year = {2025},
doi = {10.15586/aei.v53i4.1353}
}
EUVIMED