EUVIMEDEUVIMED entdecken
Open Access

PMC-LLaMA: toward building open-source language models for medicine

Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, Yanfeng Wang

Journal of the American Medical Informatics Association · 01.09.2024 · 31 (9) · 1833-1843

DOI: 10.1093/jamia/ocae045 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 13.04.2024. Verfasst von Chaoyi Wu und 5 weiteren Autor:innen. Beteiligte Einrichtung: Cooperative Medianet Innovation Center (CMIC), Shanghai Jiao Tong University, Shanghai, 200240, China. Hauptthemen laut MEDLINE: Natural Language Processing. Die Arbeit wurde bisher 321-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Science and Technology Commission of Shanghai Municipality, Higher Education Discipline, State Key Laboratory of UHD Video and Audio Production and Presentation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Recently, large language models (LLMs) have showcased remarkable capabilities in natural language understanding. While demonstrating proficiency in everyday conversations and question-answering (QA) situations, these models frequently struggle in domains that require precision, such as medical applications, due to their lack of domain-specific knowledge. In this article, we describe the procedure for building a powerful, open-source language model specifically designed for medicine applications, termed as PMC-LLaMA. Materials and methods We adapt a general-purpose LLM toward the medical domain, involving data-centric knowledge injection through the integration of 4.8M biomedical academic papers and 30K medical textbooks, as well as comprehensive domain-specific instruction fine-tuning, encompassing medical QA, rationale for reasoning, and conversational dialogues with 202M tokens.

Results

While evaluating various public medical QA benchmarks and manual rating, our lightweight PMC-LLaMA, which consists of only 13B parameters, exhibits superior performance, even surpassing ChatGPT. All models, codes, and datasets for instruction tuning will be released to the research community.

Discussion

Our contributions are 3-fold: (1) we build up an open-source LLM toward the medical domain. We believe the proposed PMC-LLaMA model can promote further development of foundation models in medicine, serving as a medical trainable basic generative language backbone; (2) we conduct thorough ablation studies to demonstrate the effectiveness of each proposed component, demonstrating how different training data and model scales affect medical LLMs; (3) we contribute a large-scale, comprehensive dataset for instruction tuning.

Conclusion

In this article, we systematically investigate the process of building up an open-source medical-specific LLM, PMC-LLaMA.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language Processing

Stichwörter der Autor:innen

ChatGPTbiomedical NLPgenerative language modelslarge language models

Autor:innen und Institutionen

  • Cooperative Medianet Innovation Center (CMIC), Shanghai Jiao Tong University, Shanghai, 200240, China.
  • Shanghai AI Laboratory, Shanghai, 200232, China.

Förderung

  • Science and Technology Commission of Shanghai Municipality
  • Higher Education Discipline
  • State Key Laboratory of UHD Video and Audio Production and Presentation
  • National Key R&D Program of China
  • Higher Education Discipline Innovation Project 111
  • State Key Laboratory of UHD Video and Audio Production and Presentation.

Zitieren

APA

Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., & Wang, Y. (2024). PMC-LLaMA: toward building open-source language models for medicine. Journal of the American Medical Informatics Association, 31(9), 1833-1843. https://doi.org/10.1093/jamia/ocae045

Vancouver

Wu C, Lin W, Zhang X, Zhang Y, Xie W, Wang Y. PMC-LLaMA: toward building open-source language models for medicine. Journal of the American Medical Informatics Association. 2024;31(9):1833-1843. doi:10.1093/jamia/ocae045

BibTeX
@article{Wu2024,
  author = {Wu, Chaoyi and Lin, Weixiong and Zhang, Xiaoman and Zhang, Ya and Xie, Weidi and Wang, Yanfeng},
  title = {PMC-LLaMA: toward building open-source language models for medicine},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  volume = {31},
  number = {9},
  pages = {1833--1843},
  doi = {10.1093/jamia/ocae045}
}
Kostenlos starten →