EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

A comparative analysis of privacy-preserving large language models for automated echocardiography report analysis

Elham Mahmoudi, Sanaz Vahdati, Chieh-Ju Chao, Bardia Khosravi, Ajay Misra, Francisco Lopez-Jimenez iD, Bradley J Erickson

Journal of the American Medical Informatics Association · 2025

DOI: 10.1093/jamia/ocaf056 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of the American Medical Informatics Association, 2025. Verfasst von Elham Mahmoudi und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, Radiology Informatics Lab, Mayo Clinic, Rochester, MN 55905, United States. Hauptthemen laut MEDLINE: Echocardiography, Natural Language Processing, Heart Valve Diseases, Confidentiality. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Automated data extraction from echocardiography reports could facilitate large-scale registry creation and clinical surveillance of valvular heart diseases (VHD). We evaluated the performance of open-source large language models (LLMs) guided by prompt instructions and chain of thought (CoT) for this task.

Methods

From consecutive transthoracic echocardiographies performed in our center, we utilized 200 random reports from 2019 for prompt optimization and 1000 from 2023 for evaluation. Five instruction-tuned LLMs (Qwen2.0-72B, Llama3.0-70B, Mixtral8-46.7B, Llama3.0-8B, and Phi3.0-3.8B) were guided by prompt instructions with and without CoT to classify prosthetic valve presence and VHD severity. Performance was evaluated using classification metrics against expert-labeled ground truth. Mean squared error (MSE) was also calculated for predicted severity’s deviation from actual severity.

Results

With CoT prompting, Llama3.0-70B and Qwen2.0 achieved the highest performance (accuracy: 99.1% and 98.9% for VHD severity; 100% and 99.9% for prosthetic valve; MSE: 0.02 and 0.05, respectively). Smaller models showed lower accuracy for VHD severity (54.1%-85.9%) but maintained high accuracy for prosthetic valve detection (>96%). Chain of thought reasoning yielded higher accuracy for larger models while increasing processing time from 2-25 to 67-154 seconds per report. Based on CoT reasonings, the wrong predictions were mainly due to model outputs being influenced by irrelevant information in the text or failure to follow the prompt instructions.

Conclusions

Our study demonstrates the near-perfect performance of open-source LLMs for automated echocardiography report interpretation with the purpose of registry formation and disease surveillance. While larger models achieved exceptional accuracy through prompt optimization, practical implementation requires balancing performance with computational efficiency.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

EchocardiographyNatural Language ProcessingHeart Valve DiseasesConfidentialityInformation Storage and RetrievalProgramming LanguagesHumansLarge Language Models

Stichwörter der Autor:innen

artificial intelligenceinformation storage and retrievalnatural language processingtransthoracic echocardiographyvalvular heart disease

Autor:innen und Institutionen

  • Department of Radiology, Radiology Informatics Lab, Mayo Clinic, Rochester, MN 55905, United States.
  • Department of Cardiovascular Medicine, Mayo Clinic Rochester, Rochester, MN 55905, United States.

ORCID-Profile: Francisco Lopez-Jimenez

Zitieren

APA

Mahmoudi, E., Vahdati, S., Chao, C. J., Khosravi, B., Misra, A., Lopez-Jimenez, F., & Erickson, B. J. (2025). A comparative analysis of privacy-preserving large language models for automated echocardiography report analysis. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocaf056

Vancouver

Mahmoudi E, Vahdati S, Chao CJ, Khosravi B, Misra A, Lopez-Jimenez F, et al. A comparative analysis of privacy-preserving large language models for automated echocardiography report analysis. Journal of the American Medical Informatics Association. 2025. doi:10.1093/jamia/ocaf056

BibTeX
@article{Mahmoudi2025,
  author = {Mahmoudi, Elham and Vahdati, Sanaz and Chao, Chieh-Ju and Khosravi, Bardia and Misra, Ajay and Lopez-Jimenez, Francisco and Erickson, Bradley J},
  title = {A comparative analysis of privacy-preserving large language models for automated echocardiography report analysis},
  journal = {Journal of the American Medical Informatics Association},
  year = {2025},
  doi = {10.1093/jamia/ocaf056}
}
Kostenlos starten →