Evaluation of the Performance of a Large Language Model to Extract Signs and Symptoms from Clinical Notes
Studies in Health Technology and Informatics · 2025
DOI: 10.3233/shti250051 ↗Kurz gefasst
Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von C Mahony Reategui-Rivera und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, Utah. Hauptthemen laut MEDLINE: Natural Language Processing, Electronic Health Records, Data Mining, Symptom Assessment. Die Arbeit wurde bisher 5-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Large language models (LLMs) have increasingly been used to extract critical information from unstructured clinical notes, which often include important details not captured in the structured sections of electronic health records (EHRs). This study assesses the performance of the GPT-4o LLM in extracting signs and symptoms (S&S) from clinical notes, focusing on both general and organ-specific (urological and cardiorespiratory) contexts. Clinical notes from the MTSamples corpora were manually annotated for comparison with the S&S extraction results using LLM. GPT-4o was applied to extract S&S using named entity recognition techniques. Key performance metrics—precision, recall, and F1-score—were used to evaluate and compare general and organ-specific results. The model showed high precision in general S&S extraction (78%) and achieved the highest precision for organ-specific tasks in the cardiorespiratory dataset (87%). For the urinary dataset, precision was also strong (81%), with balanced recall and F1-scores across analyses. These findings underscore GPT-4o’s effectiveness in both general and domain-specific S&S extraction but highlight the need for domain-specific tuning and optimization to further improve recall and generalizability in specialized medical contexts.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, Utah.
Zitieren
APA
Reategui-Rivera, C. M., & Finkelstein, J. (2025). Evaluation of the Performance of a Large Language Model to Extract Signs and Symptoms from Clinical Notes. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti250051
Vancouver
Reategui-Rivera CM, Finkelstein J. Evaluation of the Performance of a Large Language Model to Extract Signs and Symptoms from Clinical Notes. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti250051
BibTeX
@article{ReateguiRivera2025,
author = {Reategui-Rivera, C Mahony and Finkelstein, Joseph},
title = {Evaluation of the Performance of a Large Language Model to Extract Signs and Symptoms from Clinical Notes},
journal = {Studies in Health Technology and Informatics},
year = {2025},
doi = {10.3233/shti250051}
}
EUVIMED