EUVIMEDEUVIMED entdecken
Crossref

Identifying signs and symptoms of urinary tract infection from emergency department clinical notes using large language models

Mark Iscoe iD, Vimig Socrates, Aidan Gilson, Ling Chi, Huan Li, Thomas Huang, Thomas Kearns, Rachelle Perkins, Laura Khandjian, R Andrew Taylor iD

Academic Emergency Medicine · 2024-06 · 31 (6) · 599-610

DOI: 10.1111/acem.14883 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Academic Emergency Medicine, online erschienen am 03.04.2024. Verfasst von Mark Iscoe und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut, USA. Hauptthemen laut MEDLINE: Urinary Tract Infections, Natural Language Processing, Emergency Service, Hospital, Electronic Health Records. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 70 Literaturangaben. Gefördert von: Yale School of Medicine.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Natural language processing (NLP) tools including recently developed large language models (LLMs) have myriad potential applications in medical care and research, including the efficient labeling and classification of unstructured text such as electronic health record (EHR) notes. This opens the door to large‐scale projects that rely on variables that are not typically recorded in a structured form, such as patient signs and symptoms.

Objectives

This study is designed to acquaint the emergency medicine research community with the foundational elements of NLP, highlighting essential terminology, annotation methodologies, and the intricacies involved in training and evaluating NLP models. Symptom characterization is critical to urinary tract infection (UTI) diagnosis, but identification of symptoms from the EHR has historically been challenging, limiting large‐scale research, public health surveillance, and EHR‐based clinical decision support. We therefore developed and compared two NLP models to identify UTI symptoms from unstructured emergency department (ED) notes.

Methods

The study population consisted of patients aged ≥ 18 who presented to an ED in a northeastern U.S. health system between June 2013 and August 2021 and had a urinalysis performed. We annotated a random subset of 1250 ED clinician notes from these visits for a list of 17 UTI symptoms. We then developed two task‐specific LLMs to perform the task of named entity recognition: a convolutional neural network‐based model (SpaCy) and a transformer‐based model designed to process longer documents (Clinical Longformer). Models were trained on 1000 notes and tested on a holdout set of 250 notes. We compared model performance (precision, recall, F1 measure) at identifying the presence or absence of UTI symptoms at the note level.

Results

A total of 8135 entities were identified in 1250 notes; 83.6% of notes included at least one entity. Overall F1 measure for note‐level symptom identification weighted by entity frequency was 0.84 for the SpaCy model and 0.88 for the Longformer model. F1 measure for identifying presence or absence of any UTI symptom in a clinical note was 0.96 (232/250 correctly classified) for the SpaCy model and 0.98 (240/250 correctly classified) for the Longformer model.

Conclusions

The study demonstrated the utility of LLMs and transformer‐based models in particular for extracting UTI symptoms from unstructured ED clinical notes; models were highly accurate for detecting the presence or absence of any UTI symptom on the note level, with variable performance for individual symptoms.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Urinary Tract InfectionsNatural Language ProcessingEmergency Service, HospitalElectronic Health RecordsHumansMaleFemaleAdultMiddle AgedAged

Stichwörter der Autor:innen

emergency medicineinfectious diseasesinformaticslarge language modelsnamed entity recognitionnatural language processingurinary tract infection

Autor:innen und Institutionen

  • Department of Emergency Medicine, Yale School of Medicine, New Haven, Connecticut, USA.
  • Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, Connecticut, USA.
  • Program of Computational Biology and Bioinformatics, Yale University, New Haven, Connecticut, USA.
  • Yale School of Medicine, New Haven, Connecticut, USA.
  • Department of Biostatistics, Yale School of Public Health, New Haven, Connecticut, USA.

ORCID-Profile: Mark Iscoe, R Andrew Taylor

Förderung

  • Yale School of Medicine

Zitieren

APA

Iscoe, M., Socrates, V., Gilson, A., Chi, L., Li, H., Huang, T., Kearns, T., Perkins, R., Khandjian, L., & Taylor, R. A. (2024). Identifying signs and symptoms of urinary tract infection from emergency department clinical notes using large language models. Academic Emergency Medicine, 31(6), 599-610. https://doi.org/10.1111/acem.14883

Vancouver

Iscoe M, Socrates V, Gilson A, Chi L, Li H, Huang T, et al. Identifying signs and symptoms of urinary tract infection from emergency department clinical notes using large language models. Academic Emergency Medicine. 2024;31(6):599-610. doi:10.1111/acem.14883

BibTeX
@article{Iscoe2024,
  author = {Iscoe, Mark and Socrates, Vimig and Gilson, Aidan and Chi, Ling and Li, Huan and Huang, Thomas and Kearns, Thomas and Perkins, Rachelle and Khandjian, Laura and Taylor, R Andrew},
  title = {Identifying signs and symptoms of urinary tract infection from emergency department clinical notes using large language models},
  journal = {Academic Emergency Medicine},
  year = {2024},
  volume = {31},
  number = {6},
  pages = {599--610},
  doi = {10.1111/acem.14883}
}
Kostenlos starten →