Dynamic few-shot prompting for clinical note section classification using lightweight, open-source large language models
Journal of the American Medical Informatics Association · 01.07.2025 · 32 (7) · 1164-1173
DOI: 10.1093/jamia/ocaf084 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 03.06.2025. Verfasst von Kurt Miller und 6 weiteren Autor:innen. Beteiligte Einrichtung: Bioinformatics and Computational Biology Program, University of Minnesota, Rochester, MN, United States. Hauptthemen laut MEDLINE: Natural Language Processing, Electronic Health Records, Information Storage and Retrieval. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 51 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NLM NIH HHS, NIH HHS, National Institutes of Health.
Abstract (Originalsprache)
Objective
Unlocking clinical information embedded in clinical notes has been hindered to a significant degree by domain-specific and context-sensitive language. Identification of note sections and structural document elements has been shown to improve information extraction and dependent downstream clinical natural language processing (NLP) tasks and applications. This study investigates the viability of a dynamic example selection prompting method to section classification using lightweight, open-source large language models (LLMs) as a practical solution for real-world healthcare clinical NLP systems. Materials and Methods We develop a dynamic few-shot prompting approach to classifying sections where section samples are first embedded using a transformer-based model and deposited in a vector store. During inference, the embedded samples with the most similar contextual embeddings to a given input section text are retrieved from the vector store and inserted into the LLM prompt. We evaluate this technique on two datasets comprising two section schemas, including varying levels of context. We compare the performance to baseline zero-shot and randomly selected few-shot scenarios.
Results
The dynamic few-shot prompting experiments yielded the highest F1 scores in each of the classification tasks and datasets for all seven of the LLMs included in the evaluation, averaging a macro F1 increase of 39.3% and 21.1% in our primary section classification task over the zero-shot and static few-shot baselines, respectively. Discussion and Conclusion Our results showcase substantial performance improvements imparted by dynamically selecting examples for few-shot LLM prompting, and further improvement by including section context, demonstrating compelling potential for clinical applications.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Bioinformatics and Computational Biology Program, University of Minnesota, Rochester, MN, United States.
- Center for Digital Health, Mayo Clinic, Rochester, MN, United States.
- Department of Medical Informatics & Clinical Epidemiology, Oregon Health and Science University, Portland, OR, United States.
- McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX, United States.
ORCID-Profile: William Hersh, Hongfang Liu
Förderung
- NLM NIH HHS
- NIH HHS
- National Institutes of Health
Zitieren
APA
Miller, K., Bedrick, S., Lu, Q., Wen, A., Hersh, W., Roberts, K., & Liu, H. (2025). Dynamic few-shot prompting for clinical note section classification using lightweight, open-source large language models. Journal of the American Medical Informatics Association, 32(7), 1164-1173. https://doi.org/10.1093/jamia/ocaf084
Vancouver
Miller K, Bedrick S, Lu Q, Wen A, Hersh W, Roberts K, et al. Dynamic few-shot prompting for clinical note section classification using lightweight, open-source large language models. Journal of the American Medical Informatics Association. 2025;32(7):1164-1173. doi:10.1093/jamia/ocaf084
BibTeX
@article{Miller2025,
author = {Miller, Kurt and Bedrick, Steven and Lu, Qiuhao and Wen, Andrew and Hersh, William and Roberts, Kirk and Liu, Hongfang},
title = {Dynamic few-shot prompting for clinical note section classification using lightweight, open-source large language models},
journal = {Journal of the American Medical Informatics Association},
year = {2025},
volume = {32},
number = {7},
pages = {1164--1173},
doi = {10.1093/jamia/ocaf084}
}
EUVIMED