Evaluating the effectiveness of biomedical fine-tuning for large language models on clinical tasks
Journal of the American Medical Informatics Association · 01.06.2025 · 32 (6) · 1015-1024
DOI: 10.1093/jamia/ocaf045 ↗Kurz gefasst
Evaluationsstudie in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 07.04.2025. Verfasst von Felix J Dorfner und 9 weiteren Autor:innen. Beteiligte Einrichtung: Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin 10117, Germany. Hauptthemen laut MEDLINE: Natural Language Processing, Information Storage and Retrieval, Data Mining. Die Arbeit wurde bisher 53-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Federal Ministry of Education and Research, Technical University of Munich, Institute for Advanced Study.
Abstract (Originalsprache)
Objectives
Large language models (LLMs) have shown potential in biomedical applications, leading to efforts to fine-tune them on domain-specific data. However, the effectiveness of this approach remains unclear. This study aims to critically evaluate the performance of biomedically fine-tuned LLMs against their general-purpose counterparts across a range of clinical tasks. Materials and Methods We evaluated the performance of biomedically fine-tuned LLMs against their general-purpose counterparts on clinical case challenges from NEJM and JAMA, and on multiple clinical tasks, such as information extraction, document summarization and clinical coding. We used a diverse set of benchmarks specifically chosen to be outside the likely fine-tuning datasets of biomedical models, ensuring a fair assessment of generalization capabilities.
Results
Biomedical LLMs generally underperformed compared to general-purpose models, especially on tasks not focused on probing medical knowledge. While on the case challenges, larger biomedical and general-purpose models showed similar performance (eg, OpenBioLLM-70B: 66.4% vs Llama-3-70B-Instruct: 65% on JAMA), smaller biomedical models showed more pronounced underperformance (OpenBioLLM-8B: 30% vs Llama-3-8B-Instruct: 64.3% on NEJM). Similar trends appeared across CLUE benchmarks, with general-purpose models often achieving higher scores in text generation, question answering, and coding. Notably, biomedical LLMs also showed a higher tendency to hallucinate.
Discussion
Our findings challenge the assumption that biomedical fine-tuning inherently improves LLM performance, as general-purpose models consistently performed better on unseen medical tasks. Retrieval-augmented generation may offer a more effective strategy for clinical adaptation.
Conclusion
Fine-tuning LLMs on biomedical data may not yield the anticipated benefits. Alternative approaches, such as retrieval augmentation, should be further explored for effective and reliable clinical integration of LLMs.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin 10117, Germany.
- Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital and Harvard Medical School, Charlestown, MA 02129, United States.
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), Essen 45131, Germany.
- Department of Radiology, Klinikum Rechts Der Isar, Technical University Munich, Munich 81675, Germany.
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen 52074, Germany.
- Cancer Research Center Cologne Essen (CCCE), West German Cancer Center Essen, University Hospital Essen (AöR), Essen 45147, Germany.
ORCID-Profile: Felix Busch, Madhumita Sushil
Förderung
- Federal Ministry of Education and Research
- Technical University of Munich
- Institute for Advanced Study
- Free State of Bavaria
- Excellence Strategy of the Federal Government
- Länder
- Technical University of Munich - Institute for Advanced Study
Zitieren
APA
Dorfner, F. J., Dada, A., Busch, F., Makowski, M. R., Han, T., Truhn, D., Kleesiek, J., Sushil, M., Adams, L. C., & Bressem, K. K. (2025). Evaluating the effectiveness of biomedical fine-tuning for large language models on clinical tasks. Journal of the American Medical Informatics Association, 32(6), 1015-1024. https://doi.org/10.1093/jamia/ocaf045
Vancouver
Dorfner FJ, Dada A, Busch F, Makowski MR, Han T, Truhn D, et al. Evaluating the effectiveness of biomedical fine-tuning for large language models on clinical tasks. Journal of the American Medical Informatics Association. 2025;32(6):1015-1024. doi:10.1093/jamia/ocaf045
BibTeX
@article{Dorfner2025,
author = {Dorfner, Felix J and Dada, Amin and Busch, Felix and Makowski, Marcus R and Han, Tianyu and Truhn, Daniel and Kleesiek, Jens and Sushil, Madhumita and Adams, Lisa C and Bressem, Keno K},
title = {Evaluating the effectiveness of biomedical fine-tuning for large language models on clinical tasks},
journal = {Journal of the American Medical Informatics Association},
year = {2025},
volume = {32},
number = {6},
pages = {1015--1024},
doi = {10.1093/jamia/ocaf045}
}
EUVIMED