EUVIMEDEUVIMED entdecken
Leitlinie

Evaluating large language models for WAO/EAACI guideline compliance in hereditary angioedema management

Mehmet Emin Gerek, Tuğba Önalan, Fatih Çölkesen, Şevket Arslan

Allergologia et Immunopathologia · 2025

DOI: 10.15586/aei.v53i4.1353 ↗

Kurz gefasst

Leitlinie in der Zeitschrift Allergologia et Immunopathologia, 2025. Verfasst von Mehmet Emin Gerek und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Internal Medicine, Division of Clinical Immunology and Allergy, Necmettin Erbakan University, Faculty of Medicine, Konya, Türkiye; drme…. Hauptthemen laut MEDLINE: Angioedemas, Hereditary, Practice Guidelines as Topic, Guideline Adherence. Die Arbeit wurde bisher 2-mal zitiert (Crossref). Gefördert von: None..

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

Hereditary angioedema (HAE) is a rare but potentially life-threatening disorder characterized by recurrent swelling episodes. Adherence to clinical guidelines, such as the World Allergy Organization/European Academy of Allergy & Clinical Immunology (WAO/EAACI) guidelines, is crucial for effective management. With the increasing role of artificial intelligence in medicine, large language models (LLMs) offer potential for clinical decision support. This study evaluates the performance of ChatGPT, Gemini, Perplexity, and Copilot in providing guideline-adherent responses for HAE management.

Methods

Twenty-eight key recommendations from the WAO/EAACI HAE guidelines were reformulated into interrogative formats and posed to the selected LLMs. Two independent clinicians assessed responses based on accuracy, adequacy, clarity, and citation reliability using a five-point Likert scale. References were categorized as guideline-based, trustworthy, or untrustworthy. A reevaluation with explicit citation instructions was conducted, with discrepancies resolved by a third reviewer.

Results

ChatGPT and Gemini outperformed Perplexity and Copilot, achieving median accuracy and adequacy scores of 5.0 versus 3.0, respectively. ChatGPT had the lowest rate of unreliable references, whereas Gemini showed inconsistency in citation behavior. Significant differences in response quality were observed among models (p < 0.001). Providing explicit sourcing instructions improved performance consistency, particularly for Gemini.

Conclusion

ChatGPT and Gemini demonstrated superior adherence to WAO/EAACI guidelines, suggesting that LLMs can support clinical decision-making in rare diseases. However, inconsistencies in citation practices highlight the need for further validation and optimization to enhance reliability in medical applications.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Angioedemas, HereditaryPractice Guidelines as TopicGuideline AdherenceHumansLanguageArtificial IntelligenceLarge Language Models

Stichwörter der Autor:innen

WAO/EAACI guidelinesartificial Intelligencehereditary angioedemalarge language modelsmedical guidelines compliance

Autor:innen und Institutionen

  • Department of Internal Medicine, Division of Clinical Immunology and Allergy, Necmettin Erbakan University, Faculty of Medicine, Konya, Türkiye; drmegerek@gmail.com.
  • Department of Internal Medicine, Division of Clinical Immunology and Allergy, Necmettin Erbakan University, Faculty of Medicine, Konya, Türkiye.

Förderung

  • None.

Zitieren

APA

Gerek, M. E., Önalan, T., Çölkesen, F., & Arslan, Ş. (2025). Evaluating large language models for WAO/EAACI guideline compliance in hereditary angioedema management. Allergologia et Immunopathologia. https://doi.org/10.15586/aei.v53i4.1353

Vancouver

Gerek ME, Önalan T, Çölkesen F, Arslan Ş. Evaluating large language models for WAO/EAACI guideline compliance in hereditary angioedema management. Allergologia et Immunopathologia. 2025. doi:10.15586/aei.v53i4.1353

BibTeX
@article{Gerek2025,
  author = {Gerek, Mehmet Emin and Önalan, Tuğba and Çölkesen, Fatih and Arslan, Şevket},
  title = {Evaluating large language models for WAO/EAACI guideline compliance in hereditary angioedema management},
  journal = {Allergologia et Immunopathologia},
  year = {2025},
  doi = {10.15586/aei.v53i4.1353}
}
Kostenlos starten →