EchoLLM: extracting echocardiogram entities with light-weight, open-source large language models
JAMIA Open · 03.07.2025 · 8 (4)
DOI: 10.1093/jamiaopen/ooaf092 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 13.08.2025. Verfasst von Jonathan Chi und 14 weiteren Autor:innen. Beteiligte Einrichtung: Goergen Institute for Data Science and Artificial Intelligence, University of Rochester, Rochester, NY 14627, United States. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: NHLBI NIH HHS, National Institutes of Health, Washington University Institute of Clinical and Translational Sciences.
Abstract (Originalsprache)
Objectives
Large language models (LLMs) have demonstrated high levels of performance in clinical information extraction compared to rule-based systems and traditional machine-learning approaches, offering scalability, contextualization, and easier deployment. However, most studies rely on proprietary models with privacy concerns and high costs, limiting accessibility. We aim to evaluate 14 publicly available open-source LLMs for extracting clinically relevant findings from free-text echocardiogram reports and examine the feasibility of their implementation in information extraction workflows. Materials and Methods We used 14 open-source LLM models to extract clinically relevant entities from echocardiogram reports (n = 507). Each report was manually annotated by 2 independent health-care professionals and adjudicated by a third. Lexical variance and length of each echocardiogram report were collected. Precision, recall, and F1 scores were calculated for the 9 extracted entities via multiclass classification.
Results
In aggregate, Gemma2:9b-instruct had the highest precision, recall, and F1 scores at 0.973 (0.962-0.983), 0.959 (0.947-0.973), and 0.965 (0.951-0.975), respectively. In comparison, Phi3:3.8b-mini-instruct had the lowest precision score at 0.831 (0.804-0.856), while Gemma:7b-instruct had the lowest recall and F1 scores at 0.382 (0.356-0.408) and 0.392 (0.356-0.428), respectively. Discussion and Conclusion Using LLMs for entity extraction for echocardiogram reports has the potential to support both clinical research and health-care delivery. Our work demonstrates the feasibility of using open-source models for more efficient computation and extraction.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Goergen Institute for Data Science and Artificial Intelligence, University of Rochester, Rochester, NY 14627, United States.
- Department of Medicine, Institute for Informatics, Data Science and Biostatistics, Washington University in St. Louis, St. Louis, MO 63110, United States.
- Division of Hospital Medicine, Department of Medicine, Washington University in St. Louis, St. Louis, MO 63110, United States.
- Division of Pulmonary and Critical Care Medicine, Department of Medicine, Washington University in St. Louis, St. Louis, MO 63110, United States.
- Division of Pulmonary, Allergy, and Critical Care Medicine, Department of Medicine, Oregon Health & Science University, Portland, OR 97239, United States.
- School of Nursing, University of Rochester, Rochester, NY 14627, United States.
ORCID-Profile: Jonathan Chi, Yazan Rouphail, Mackenzie Hofford, Patrick G Lyons, Albert M Lai, Philip R O Payne, Caitlin Dreisbach
Förderung
- NHLBI NIH HHS
- National Institutes of Health
- Washington University Institute of Clinical and Translational Sciences
Zitieren
APA
Chi, J., Rouphail, Y., Hillis, E., Ma, N., Nguyen, A., Wang, J., Hofford, M., Gupta, A., Lyons, P. G., Wilcox, A., Lai, A. M., Payne, P. R. O., Kollef, M. H., Dreisbach, C., & Michelson, A. P. (2025). EchoLLM: extracting echocardiogram entities with light-weight, open-source large language models. JAMIA Open, 8(4). https://doi.org/10.1093/jamiaopen/ooaf092
Vancouver
Chi J, Rouphail Y, Hillis E, Ma N, Nguyen A, Wang J, et al. EchoLLM: extracting echocardiogram entities with light-weight, open-source large language models. JAMIA Open. 2025;8(4). doi:10.1093/jamiaopen/ooaf092
BibTeX
@article{Chi2025,
author = {Chi, Jonathan and Rouphail, Yazan and Hillis, Ethan and Ma, Ningning and Nguyen, An and Wang, Jane and Hofford, Mackenzie and Gupta, Aditi and Lyons, Patrick G and Wilcox, Adam and Lai, Albert M and Payne, Philip R O and Kollef, Marin H and Dreisbach, Caitlin and Michelson, Andrew P},
title = {EchoLLM: extracting echocardiogram entities with light-weight, open-source large language models},
journal = {JAMIA Open},
year = {2025},
volume = {8},
number = {4},
doi = {10.1093/jamiaopen/ooaf092}
}
EUVIMED