EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Utilizing large language models for detecting hospital-acquired conditions: an empirical study on pulmonary embolism

Cheligeer Cheligeer iD, Danielle A Southern iD, Jun Yan iD, Guosong Wu, Jie Pan iD, Seungwon Lee, Elliot A Martin iD, Hamed Jafarpour iD, Cathy A Eastwood iD, Yong Zeng iD, Hude Quan iD

Journal of the American Medical Informatics Association · 2025

DOI: 10.1093/jamia/ocaf048 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2025. Verfasst von Cheligeer Cheligeer und 10 weiteren Autor:innen. Beteiligte Einrichtung: Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary T2N 4N1, Canada. Hauptthemen laut MEDLINE: Pulmonary Embolism, Electronic Health Records, Natural Language Processing, Iatrogenic Disease. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIDCD NIH HHS, CIHR, Canadian Institutes of Health Research.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Adverse event detection from Electronic Medical Records (EMRs) is challenging due to the low incidence of the event, variability in clinical documentation, and the complexity of data formats. Pulmonary embolism as an adverse event (PEAE) is particularly difficult to identify using existing approaches. This study aims to develop and evaluate a Large Language Model (LLM)-based framework for detecting PEAE from unstructured narrative data in EMRs. Materials and Methods We conducted a chart review of adult patients (aged 18-100) admitted to tertiary-care hospitals in Calgary, Alberta, Canada, between 2017-2022. We developed an LLM-based detection framework consisting of three modules: evidence extraction (implementing both keyword-based and semantic similarity-based filtering methods), discharge information extraction (focusing on six key clinical sections), and PEAE detection. Four open-source LLMs (Llama3, Mistral-7B, Gemma, and Phi-3) were evaluated using positive predictive value, sensitivity, specificity, and F1-score. Model performance for population-level surveillance was assessed at yearly, quarterly, and monthly granularities.

Results

The chart review included 10 066 patients, with 40 cases of PEAE identified (0.4% prevalence). All four LLMs demonstrated high sensitivity (87.5-100%) and specificity (94.9-98.9%) across different experimental conditions. Gemma achieved the highest F1-score (28.11%) using keyword-based retrieval with discharge summary inclusion, along with 98.4% specificity, 87.5% sensitivity, and 99.95% negative predictive value. Keyword-based filtering reduced the median chunks per patient from 789 to 310, while semantic filtering further reduced this to 9 chunks. Including discharge summaries improved performance metrics across most models. For population-level surveillance, all models showed strong correlation with actual PEAE trends at yearly granularity (r=0.92-0.99), with Llama3 achieving the highest correlation (0.988).

Discussion

The results of our method for PEAE detection using EMR notes demonstrate high sensitivity and specificity across all four tested LLMs, indicating strong performance in distinguishing PEAE from non-PEAE cases. However, the low incidence rate of PEAE contributed to a lower PPV. The keyword-based chunking approach consistently outperformed semantic similarity-based methods, achieving higher F1 scores and PPV, underscoring the importance of domain knowledge in text segmentation. Including discharge summaries further enhanced performance metrics. Our population-based analysis revealed better performance for yearly trends compared to monthly granularity, suggesting the framework's utility for long-term surveillance despite dataset imbalance. Error analysis identified contextual misinterpretation, terminology confusion, and preprocessing limitations as key challenges for future improvement.

Conclusions

Our proposed method demonstrates that LLMs can effectively detect PEAE from narrative EMRs with high sensitivity and specificity. While these models serve as effective screening tools to exclude non-PEAE cases, their lower PPV indicates they cannot be relied upon solely for definitive PEAE identification. Further chart review remains necessary for confirmation. Future work should focus on improving contextual understanding, medical terminology interpretation, and exploring advanced prompting techniques to enhance precision in adverse event detection from EMRs.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Pulmonary EmbolismElectronic Health RecordsNatural Language ProcessingIatrogenic DiseaseHumansMiddle AgedAdultAgedAged, 80 and overAlbertaAdolescentYoung AdultMaleFemaleSensitivity and SpecificityLarge Language Models

Stichwörter der Autor:innen

adverse event detectionclinical text mininglarge language modelspulmonary embolism

Autor:innen und Institutionen

  • Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary T2N 4N1, Canada.
  • Provincial Research Data Services, Alberta Health Services, Calgary T2N 4N1, Canada.
  • Concordia Institute for Information Systems Engineering, Concordia University, Montreal H3G 2W1, Canada.
  • Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary T2N 4N1, Canada.
  • Libin Cardiovascular Institute, University of Calgary, Calgary T2N 4N1, Canada.

ORCID-Profile: Cheligeer Cheligeer, Danielle A Southern, Jun Yan, Jie Pan, Elliot A Martin, Hamed Jafarpour, Cathy A Eastwood, Yong Zeng, Hude Quan

Förderung

  • NIDCD NIH HHS
  • CIHR
  • Canadian Institutes of Health Research

Zitieren

APA

Cheligeer, C., Southern, D. A., Yan, J., Wu, G., Pan, J., Lee, S., Martin, E. A., Jafarpour, H., Eastwood, C. A., Zeng, Y., & Quan, H. (2025). Utilizing large language models for detecting hospital-acquired conditions: an empirical study on pulmonary embolism. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocaf048

Vancouver

Cheligeer C, Southern DA, Yan J, Wu G, Pan J, Lee S, et al. Utilizing large language models for detecting hospital-acquired conditions: an empirical study on pulmonary embolism. Journal of the American Medical Informatics Association. 2025. doi:10.1093/jamia/ocaf048

BibTeX
@article{Cheligeer2025,
  author = {Cheligeer, Cheligeer and Southern, Danielle A and Yan, Jun and Wu, Guosong and Pan, Jie and Lee, Seungwon and Martin, Elliot A and Jafarpour, Hamed and Eastwood, Cathy A and Zeng, Yong and Quan, Hude},
  title = {Utilizing large language models for detecting hospital-acquired conditions: an empirical study on pulmonary embolism},
  journal = {Journal of the American Medical Informatics Association},
  year = {2025},
  doi = {10.1093/jamia/ocaf048}
}
Kostenlos starten →