PMC-LLaMA: toward building open-source language models for medicine
Journal of the American Medical Informatics Association · 01.09.2024 · 31 (9) · 1833-1843
DOI: 10.1093/jamia/ocae045 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 13.04.2024. Verfasst von Chaoyi Wu und 5 weiteren Autor:innen. Beteiligte Einrichtung: Cooperative Medianet Innovation Center (CMIC), Shanghai Jiao Tong University, Shanghai, 200240, China. Hauptthemen laut MEDLINE: Natural Language Processing. Die Arbeit wurde bisher 321-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Science and Technology Commission of Shanghai Municipality, Higher Education Discipline, State Key Laboratory of UHD Video and Audio Production and Presentation.
Abstract (Originalsprache)
Objective
Recently, large language models (LLMs) have showcased remarkable capabilities in natural language understanding. While demonstrating proficiency in everyday conversations and question-answering (QA) situations, these models frequently struggle in domains that require precision, such as medical applications, due to their lack of domain-specific knowledge. In this article, we describe the procedure for building a powerful, open-source language model specifically designed for medicine applications, termed as PMC-LLaMA. Materials and methods We adapt a general-purpose LLM toward the medical domain, involving data-centric knowledge injection through the integration of 4.8M biomedical academic papers and 30K medical textbooks, as well as comprehensive domain-specific instruction fine-tuning, encompassing medical QA, rationale for reasoning, and conversational dialogues with 202M tokens.
Results
While evaluating various public medical QA benchmarks and manual rating, our lightweight PMC-LLaMA, which consists of only 13B parameters, exhibits superior performance, even surpassing ChatGPT. All models, codes, and datasets for instruction tuning will be released to the research community.
Discussion
Our contributions are 3-fold: (1) we build up an open-source LLM toward the medical domain. We believe the proposed PMC-LLaMA model can promote further development of foundation models in medicine, serving as a medical trainable basic generative language backbone; (2) we conduct thorough ablation studies to demonstrate the effectiveness of each proposed component, demonstrating how different training data and model scales affect medical LLMs; (3) we contribute a large-scale, comprehensive dataset for instruction tuning.
Conclusion
In this article, we systematically investigate the process of building up an open-source medical-specific LLM, PMC-LLaMA.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Cooperative Medianet Innovation Center (CMIC), Shanghai Jiao Tong University, Shanghai, 200240, China.
- Shanghai AI Laboratory, Shanghai, 200232, China.
Förderung
- Science and Technology Commission of Shanghai Municipality
- Higher Education Discipline
- State Key Laboratory of UHD Video and Audio Production and Presentation
- National Key R&D Program of China
- Higher Education Discipline Innovation Project 111
- State Key Laboratory of UHD Video and Audio Production and Presentation.
Zitieren
APA
Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., & Wang, Y. (2024). PMC-LLaMA: toward building open-source language models for medicine. Journal of the American Medical Informatics Association, 31(9), 1833-1843. https://doi.org/10.1093/jamia/ocae045
Vancouver
Wu C, Lin W, Zhang X, Zhang Y, Xie W, Wang Y. PMC-LLaMA: toward building open-source language models for medicine. Journal of the American Medical Informatics Association. 2024;31(9):1833-1843. doi:10.1093/jamia/ocae045
BibTeX
@article{Wu2024,
author = {Wu, Chaoyi and Lin, Weixiong and Zhang, Xiaoman and Zhang, Ya and Xie, Weidi and Wang, Yanfeng},
title = {PMC-LLaMA: toward building open-source language models for medicine},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
volume = {31},
number = {9},
pages = {1833--1843},
doi = {10.1093/jamia/ocae045}
}
EUVIMED