EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Performance of Open-Source Large Language Models to Extract Symptoms from Clinical Notes

Yunbing Bai, Wanting Cui, Joseph Finkelstein

Studies in Health Technology and Informatics · 2025

DOI: 10.3233/shti250923 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Yunbing Bai und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, Utah. Hauptthemen laut MEDLINE: Natural Language Processing, Electronic Health Records, Data Mining, Symptom Assessment. Die Arbeit wurde bisher 3-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this study, we examined how well the open-source foundational large language models (LLMs) can extract symptoms and signs (S&S), along with their corresponding ICD-10 codes, from clinical notes found in the public MTSamples dataset. The dataset comprising notes of patients with genitourinary conditions was manually annotated to compare the S&S extraction results with outputs generated by LLMs. We assessed three versions of the Llama model—Llama 3.1-13B, Llama 3.3-70B, and Me-Llama-13B—focusing on their consistency, runtime, and performance. Each model was tested on two tasks: (1) S&S extraction and (2) ICD-10 code generation. Our findings indicate that Llama 3.3-70B performed the best overall. With fast runtime and high consistency, it achieved an average recall of 0.87 and an average precision of 0.71 for S&S extraction, as well as an average recall of 0.71 and an average precision of 0.54 for ICD-10 code generation.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingElectronic Health RecordsData MiningSymptom AssessmentHumansInternational Classification of DiseasesLarge Language Models

Stichwörter der Autor:innen

Large Language ModelsLlama ModelsNatural Language ProcessingSymptom Extraction

Autor:innen und Institutionen

  • Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, Utah.

Zitieren

APA

Bai, Y., Cui, W., & Finkelstein, J. (2025). Performance of Open-Source Large Language Models to Extract Symptoms from Clinical Notes. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti250923

Vancouver

Bai Y, Cui W, Finkelstein J. Performance of Open-Source Large Language Models to Extract Symptoms from Clinical Notes. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti250923

BibTeX
@article{Bai2025,
  author = {Bai, Yunbing and Cui, Wanting and Finkelstein, Joseph},
  title = {Performance of Open-Source Large Language Models to Extract Symptoms from Clinical Notes},
  journal = {Studies in Health Technology and Informatics},
  year = {2025},
  doi = {10.3233/shti250923}
}
Kostenlos starten →