EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Evaluation of the Performance of a Large Language Model to Extract Signs and Symptoms from Clinical Notes

C Mahony Reategui-Rivera, Joseph Finkelstein

Studies in Health Technology and Informatics · 2025

DOI: 10.3233/shti250051 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von C Mahony Reategui-Rivera und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, Utah. Hauptthemen laut MEDLINE: Natural Language Processing, Electronic Health Records, Data Mining, Symptom Assessment. Die Arbeit wurde bisher 5-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) have increasingly been used to extract critical information from unstructured clinical notes, which often include important details not captured in the structured sections of electronic health records (EHRs). This study assesses the performance of the GPT-4o LLM in extracting signs and symptoms (S&S) from clinical notes, focusing on both general and organ-specific (urological and cardiorespiratory) contexts. Clinical notes from the MTSamples corpora were manually annotated for comparison with the S&S extraction results using LLM. GPT-4o was applied to extract S&S using named entity recognition techniques. Key performance metrics—precision, recall, and F1-score—were used to evaluate and compare general and organ-specific results. The model showed high precision in general S&S extraction (78%) and achieved the highest precision for organ-specific tasks in the cardiorespiratory dataset (87%). For the urinary dataset, precision was also strong (81%), with balanced recall and F1-scores across analyses. These findings underscore GPT-4o’s effectiveness in both general and domain-specific S&S extraction but highlight the need for domain-specific tuning and optimization to further improve recall and generalizability in specialized medical contexts.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingElectronic Health RecordsData MiningSymptom AssessmentHumansLarge Language Models

Stichwörter der Autor:innen

Large Language ModelsNamed Entity RecognitionNatural Language ProcessingSigns and Symptoms

Autor:innen und Institutionen

  • Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, Utah.

Zitieren

APA

Reategui-Rivera, C. M., & Finkelstein, J. (2025). Evaluation of the Performance of a Large Language Model to Extract Signs and Symptoms from Clinical Notes. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti250051

Vancouver

Reategui-Rivera CM, Finkelstein J. Evaluation of the Performance of a Large Language Model to Extract Signs and Symptoms from Clinical Notes. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti250051

BibTeX
@article{ReateguiRivera2025,
  author = {Reategui-Rivera, C Mahony and Finkelstein, Joseph},
  title = {Evaluation of the Performance of a Large Language Model to Extract Signs and Symptoms from Clinical Notes},
  journal = {Studies in Health Technology and Informatics},
  year = {2025},
  doi = {10.3233/shti250051}
}
Kostenlos starten →