Evaluating large language models for specialist referral triage in primary care: a quantitative study using otolaryngology scenarios
Family Practice · 2025
DOI: 10.1093/fampra/cmaf087 ↗Kurz gefasst
Fachartikel in der Zeitschrift Family Practice, 2025. Verfasst von Sholem Hack und 6 weiteren Autor:innen. Beteiligte Einrichtung: City St. George's University London School of Medicine, Program Delivered by University of Nicosia at the Chaim Sheba Medical Center, Ramat Gan 52621…. Hauptthemen laut MEDLINE: Triage, Referral and Consultation, Primary Health Care, Otolaryngology. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 33 Literaturangaben.
Abstract (Originalsprache)
Background
Effective and timely referral to specialist care is a fundamental responsibility of primary care providers, including family physicians, general practitioners, and community health workers. However, challenges in triage can lead to delays, unnecessary referrals, and increased strain on healthcare systems. Advances in artificial intelligence (AI) now offer new opportunities to support referral decision-making at the primary care level.
Objective
This study evaluates the performance and reliability of two AI models as referral decision support tools, using simulated scenarios commonly encountered in primary care settings involving ear, nose, and throat conditions.
Methods
Sixteen clinical vignettes representing common or high-stakes primary care presentations requiring specialist input were presented to each model in both structured clinical and informal patient-language formats. Responses were independently assessed by five otolaryngologists and 10 lay reviewers using a standardized rubric focused on appropriateness, clarity, safety, and usefulness. Quantitative analysis included comparisons of model performance, reviewer agreement, and the impact of prompt structure on output quality.
Results
Both AI models generated safe and clinically appropriate referral recommendations when provided with structured clinical input. No statistically significant differences were observed between the two models across the evaluated domains. Performance declined for one model when prompts were presented in informal language, underscoring the importance of clear input structure. Reviewer agreement was high, confirming the reliability of findings.
Conclusion
AI decision support tools show potential to assist specialist referral triage in primary care. Clear, structured input is essential to maximize safety and reliability.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- City St. George's University London School of Medicine, Program Delivered by University of Nicosia at the Chaim Sheba Medical Center, Ramat Gan 52621, Israel.
- Department of Otolaryngology, Sheba Medical Center, Ramat Gan 52621, Israel.
- Faculty of Medicine, Tel-Aviv University, Tel Aviv 69978, Israel.
ORCID-Profile: Sholem Hack, Eran Glikson
Zitieren
APA
Hack, S., Attal, R., Yogev, D., Alsleibi, S., Gvili, B., Shahal, D., & Glikson, E. (2025). Evaluating large language models for specialist referral triage in primary care: a quantitative study using otolaryngology scenarios. Family Practice. https://doi.org/10.1093/fampra/cmaf087
Vancouver
Hack S, Attal R, Yogev D, Alsleibi S, Gvili B, Shahal D, et al. Evaluating large language models for specialist referral triage in primary care: a quantitative study using otolaryngology scenarios. Family Practice. 2025. doi:10.1093/fampra/cmaf087
BibTeX
@article{Hack2025,
author = {Hack, Sholem and Attal, Rebecca and Yogev, David and Alsleibi, Shibli and Gvili, Ben and Shahal, David and Glikson, Eran},
title = {Evaluating large language models for specialist referral triage in primary care: a quantitative study using otolaryngology scenarios},
journal = {Family Practice},
year = {2025},
doi = {10.1093/fampra/cmaf087}
}
EUVIMED