Utilizing large language models for detecting hospital-acquired conditions: an empirical study on pulmonary embolism
Journal of the American Medical Informatics Association · 2025
DOI: 10.1093/jamia/ocaf048 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2025. Verfasst von Cheligeer Cheligeer und 10 weiteren Autor:innen. Beteiligte Einrichtung: Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary T2N 4N1, Canada. Hauptthemen laut MEDLINE: Pulmonary Embolism, Electronic Health Records, Natural Language Processing, Iatrogenic Disease. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIDCD NIH HHS, CIHR, Canadian Institutes of Health Research.
Abstract (Originalsprache)
Objectives
Adverse event detection from Electronic Medical Records (EMRs) is challenging due to the low incidence of the event, variability in clinical documentation, and the complexity of data formats. Pulmonary embolism as an adverse event (PEAE) is particularly difficult to identify using existing approaches. This study aims to develop and evaluate a Large Language Model (LLM)-based framework for detecting PEAE from unstructured narrative data in EMRs. Materials and Methods We conducted a chart review of adult patients (aged 18-100) admitted to tertiary-care hospitals in Calgary, Alberta, Canada, between 2017-2022. We developed an LLM-based detection framework consisting of three modules: evidence extraction (implementing both keyword-based and semantic similarity-based filtering methods), discharge information extraction (focusing on six key clinical sections), and PEAE detection. Four open-source LLMs (Llama3, Mistral-7B, Gemma, and Phi-3) were evaluated using positive predictive value, sensitivity, specificity, and F1-score. Model performance for population-level surveillance was assessed at yearly, quarterly, and monthly granularities.
Results
The chart review included 10 066 patients, with 40 cases of PEAE identified (0.4% prevalence). All four LLMs demonstrated high sensitivity (87.5-100%) and specificity (94.9-98.9%) across different experimental conditions. Gemma achieved the highest F1-score (28.11%) using keyword-based retrieval with discharge summary inclusion, along with 98.4% specificity, 87.5% sensitivity, and 99.95% negative predictive value. Keyword-based filtering reduced the median chunks per patient from 789 to 310, while semantic filtering further reduced this to 9 chunks. Including discharge summaries improved performance metrics across most models. For population-level surveillance, all models showed strong correlation with actual PEAE trends at yearly granularity (r=0.92-0.99), with Llama3 achieving the highest correlation (0.988).
Discussion
The results of our method for PEAE detection using EMR notes demonstrate high sensitivity and specificity across all four tested LLMs, indicating strong performance in distinguishing PEAE from non-PEAE cases. However, the low incidence rate of PEAE contributed to a lower PPV. The keyword-based chunking approach consistently outperformed semantic similarity-based methods, achieving higher F1 scores and PPV, underscoring the importance of domain knowledge in text segmentation. Including discharge summaries further enhanced performance metrics. Our population-based analysis revealed better performance for yearly trends compared to monthly granularity, suggesting the framework's utility for long-term surveillance despite dataset imbalance. Error analysis identified contextual misinterpretation, terminology confusion, and preprocessing limitations as key challenges for future improvement.
Conclusions
Our proposed method demonstrates that LLMs can effectively detect PEAE from narrative EMRs with high sensitivity and specificity. While these models serve as effective screening tools to exclude non-PEAE cases, their lower PPV indicates they cannot be relied upon solely for definitive PEAE identification. Further chart review remains necessary for confirmation. Future work should focus on improving contextual understanding, medical terminology interpretation, and exploring advanced prompting techniques to enhance precision in adverse event detection from EMRs.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary T2N 4N1, Canada.
- Provincial Research Data Services, Alberta Health Services, Calgary T2N 4N1, Canada.
- Concordia Institute for Information Systems Engineering, Concordia University, Montreal H3G 2W1, Canada.
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary T2N 4N1, Canada.
- Libin Cardiovascular Institute, University of Calgary, Calgary T2N 4N1, Canada.
ORCID-Profile: Cheligeer Cheligeer, Danielle A Southern, Jun Yan, Jie Pan, Elliot A Martin, Hamed Jafarpour, Cathy A Eastwood, Yong Zeng, Hude Quan
Förderung
- NIDCD NIH HHS
- CIHR
- Canadian Institutes of Health Research
Zitieren
APA
Cheligeer, C., Southern, D. A., Yan, J., Wu, G., Pan, J., Lee, S., Martin, E. A., Jafarpour, H., Eastwood, C. A., Zeng, Y., & Quan, H. (2025). Utilizing large language models for detecting hospital-acquired conditions: an empirical study on pulmonary embolism. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocaf048
Vancouver
Cheligeer C, Southern DA, Yan J, Wu G, Pan J, Lee S, et al. Utilizing large language models for detecting hospital-acquired conditions: an empirical study on pulmonary embolism. Journal of the American Medical Informatics Association. 2025. doi:10.1093/jamia/ocaf048
BibTeX
@article{Cheligeer2025,
author = {Cheligeer, Cheligeer and Southern, Danielle A and Yan, Jun and Wu, Guosong and Pan, Jie and Lee, Seungwon and Martin, Elliot A and Jafarpour, Hamed and Eastwood, Cathy A and Zeng, Yong and Quan, Hude},
title = {Utilizing large language models for detecting hospital-acquired conditions: an empirical study on pulmonary embolism},
journal = {Journal of the American Medical Informatics Association},
year = {2025},
doi = {10.1093/jamia/ocaf048}
}
EUVIMED