Supervised Fine-Tuning of Large Language Models With Chain-of-Thought Reasoning for Pediatric Heart Disease Detection in Unstructured Echocardiogram Reports: Algorithm Development and Validation
JMIR Formative Research · 2026
DOI: 10.2196/90968 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Formative Research, 2026. Verfasst von Haoming Shi und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Engineering, Duke University, 1427 Fitzpatrick Center Box 90281, Durham, NC, 27708, United States, 1 9196695131. Hauptthemen laut MEDLINE: Large Language Models, Algorithms, Echocardiography, Heart Diseases. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Pediatric heart disease (PHD), including congenital heart defects, is often incompletely captured in electronic health records, particularly when clinical significance must be inferred from unstructured echocardiogram reports. Automated methods capable of extracting clinically meaningful PHD from narrative reports could improve clinical decision support and research applications.
Objective
The aim of the study is to evaluate the feasibility of using supervised fine-tuning of large language models (LLMs), with and without chain-of-thought (CoT) reasoning, to characterize patients with clinically significant or historical PHD from unstructured echocardiogram reports.
Methods
We developed a PHD detection algorithm using fine-tuned open-source LLMs, including LLaMA (Meta) and Qwen (Alibaba), to analyze 9749 echocardiogram reports. A subset of 712 reports was adjudicated by 2 pediatric cardiac anesthesiologists, classifying 506 (71.1%) as clinically significant PHD and 206 (28.9%) as not significant. While DeepSeek R1 has shown improved performance with CoT reasoning, its application in medical contexts is underexplored. We incorporated R1-generated CoT into model prompts and fine-tuned backbone LLMs.
Results
The fine-tuned Qwen-7B-10k-overthink-CoT achieved the highest accuracy (92.4%), outperforming Qwen-7B-without-CoT (90%), LLaMA-3B-without-CoT (87.9%), Qwen-3B-without-CoT (85.6%), Qwen-3B-10k-overthink-CoT (68.5%), and LLaMA-3B-10k-overthink-CoT (46.2%). In a second dataset, an external validation was performed (n=113; 64 positive, 49 negative), Qwen-7B-10k-overthink-CoT sustained a strong, balanced performance (82.7%), followed by Qwen-7B-without-CoT (88.4%), LLaMA-3B-without-CoT (86.8%), Qwen-3B-without-CoT (84.5%), Qwen-3B-10k-overthink-CoT (58.9%), and LLaMA-3B-10k-overthink-CoT (46.2%). The fine-tuned Qwen-7B model with overthinking CoT (10,000 tokens) achieved the highest internal accuracy (92.4%), with balanced sensitivity and specificity. Across repeated runs, CoT-enhanced models demonstrated improved classification consistency compared to non-CoT models (Qwen-7B-without-CoT: 90%, LLaMA-3B-without-CoT: 87.9%, Qwen-3B-without-CoT: 85.6%). In external validation (n=113), non-CoT variants achieved higher accuracy (up to 88.4%), whereas the Qwen-7B CoT model demonstrated more balanced class performance (accuracy=82.7%).
Conclusions
Supervised fine-tuning of LLMs with CoT offers an effective approach for automated PHD detection within unstructured data in the electronic medical record. While CoT-enhanced models demonstrated improved internal performance and more balanced classification, they did not consistently achieve higher accuracy in external validation, highlighting trade-offs between accuracy and class balance. These findings highlight the promise of LLM-based approaches for clinical text phenotyping while underscoring the need for larger, multicenter validation and careful calibration for real-world deployment. Continued validation and integration into the electronic medical record are essential for real-world, artificial intelligence–driven clinical decision support.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Engineering, Duke University, 1427 Fitzpatrick Center Box 90281, Durham, NC, 27708, United States, 1 9196695131.
- Department of Surgery, Duke University School of Medicine, Durham, NC, United States.
- Department of Anesthesiology, Duke University School of Medicine, Durham, NC, United States.
- Department of Anesthesiology, Emory University School of Medicine, Atlanta, GA, United States.
- Department of Anesthesiology, Children's Healthcare of Atlanta, Atlanta, GA, United States.
- Department of Pediatrics, Duke University School of Medicine, Durham, NC, United States.
ORCID-Profile: Haoming Shi, Justin B Long, Michael C Fiedorek, Hannah D Kilday, Henry P Foote, Christoph P Hornik, Aditya Nagori, Yifan Xiang, Rishikesan Kamaleswaran
Zitieren
APA
Shi, H., Long, J. B., Fiedorek, M. C., Kilday, H. D., Foote, H. P., Hornik, C. P., Nagori, A., Xiang, Y., & Kamaleswaran, R. (2026). Supervised Fine-Tuning of Large Language Models With Chain-of-Thought Reasoning for Pediatric Heart Disease Detection in Unstructured Echocardiogram Reports: Algorithm Development and Validation. JMIR Formative Research. https://doi.org/10.2196/90968
Vancouver
Shi H, Long JB, Fiedorek MC, Kilday HD, Foote HP, Hornik CP, et al. Supervised Fine-Tuning of Large Language Models With Chain-of-Thought Reasoning for Pediatric Heart Disease Detection in Unstructured Echocardiogram Reports: Algorithm Development and Validation. JMIR Formative Research. 2026. doi:10.2196/90968
BibTeX
@article{Shi2026,
author = {Shi, Haoming and Long, Justin B and Fiedorek, Michael C and Kilday, Hannah D and Foote, Henry P and Hornik, Christoph P and Nagori, Aditya and Xiang, Yifan and Kamaleswaran, Rishikesan},
title = {Supervised Fine-Tuning of Large Language Models With Chain-of-Thought Reasoning for Pediatric Heart Disease Detection in Unstructured Echocardiogram Reports: Algorithm Development and Validation},
journal = {JMIR Formative Research},
year = {2026},
doi = {10.2196/90968}
}
EUVIMED