Performance of Open-Source Large Language Models to Extract Symptoms from Clinical Notes
Studies in Health Technology and Informatics · 2025
DOI: 10.3233/shti250923 ↗Kurz gefasst
Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Yunbing Bai und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, Utah. Hauptthemen laut MEDLINE: Natural Language Processing, Electronic Health Records, Data Mining, Symptom Assessment. Die Arbeit wurde bisher 3-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
In this study, we examined how well the open-source foundational large language models (LLMs) can extract symptoms and signs (S&S), along with their corresponding ICD-10 codes, from clinical notes found in the public MTSamples dataset. The dataset comprising notes of patients with genitourinary conditions was manually annotated to compare the S&S extraction results with outputs generated by LLMs. We assessed three versions of the Llama model—Llama 3.1-13B, Llama 3.3-70B, and Me-Llama-13B—focusing on their consistency, runtime, and performance. Each model was tested on two tasks: (1) S&S extraction and (2) ICD-10 code generation. Our findings indicate that Llama 3.3-70B performed the best overall. With fast runtime and high consistency, it achieved an average recall of 0.87 and an average precision of 0.71 for S&S extraction, as well as an average recall of 0.71 and an average precision of 0.54 for ICD-10 code generation.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, Utah.
Zitieren
APA
Bai, Y., Cui, W., & Finkelstein, J. (2025). Performance of Open-Source Large Language Models to Extract Symptoms from Clinical Notes. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti250923
Vancouver
Bai Y, Cui W, Finkelstein J. Performance of Open-Source Large Language Models to Extract Symptoms from Clinical Notes. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti250923
BibTeX
@article{Bai2025,
author = {Bai, Yunbing and Cui, Wanting and Finkelstein, Joseph},
title = {Performance of Open-Source Large Language Models to Extract Symptoms from Clinical Notes},
journal = {Studies in Health Technology and Informatics},
year = {2025},
doi = {10.3233/shti250923}
}
EUVIMED