EUVIMEDEUVIMED entdecken
Open Access

Large language models leverage external knowledge to extend clinical insight beyond language boundaries

Jiageng Wu, Xian Wu, Zhaopeng Qiu, Minghui Li, Shixu Lin, Yingying Zhang, Yefeng Zheng, Changzheng Yuan, Jie Yang

Journal of the American Medical Informatics Association · 01.09.2024 · 31 (9) · 2054-2064

DOI: 10.1093/jamia/ocae079 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 29.04.2024. Verfasst von Jiageng Wu und 8 weiteren Autor:innen. Beteiligte Einrichtung: School of Public Health, Zhejiang University School of Medicine, Hangzhou, 310058, China. Hauptthemen laut MEDLINE: Knowledge Bases, Language. Die Arbeit wurde bisher 41-mal zitiert (Crossref) und nennt 75 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Large Language Models (LLMs) such as ChatGPT and Med-PaLM have excelled in various medical question-answering tasks. However, these English-centric models encounter challenges in non-English clinical settings, primarily due to limited clinical knowledge in respective languages, a consequence of imbalanced training corpora. We systematically evaluate LLMs in the Chinese medical context and develop a novel in-context learning framework to enhance their performance. Materials and Methods The latest China National Medical Licensing Examination (CNMLE-2022) served as the benchmark. We collected 53 medical books and 381 149 medical questions to construct the medical knowledge base and question bank. The proposed Knowledge and Few-shot Enhancement In-context Learning (KFE) framework leverages the in-context learning ability of LLMs to integrate diverse external clinical knowledge sources. We evaluated KFE with ChatGPT (GPT-3.5), GPT-4, Baichuan2-7B, Baichuan2-13B, and QWEN-72B in CNMLE-2022 and further investigated the effectiveness of different pathways for incorporating LLMs with medical knowledge from 7 distinct perspectives.

Results

Directly applying ChatGPT failed to qualify for the CNMLE-2022 at a score of 51. Cooperated with the KFE framework, the LLMs with varying sizes yielded consistent and significant improvements. The ChatGPT’s performance surged to 70.04 and GPT-4 achieved the highest score of 82.59. This surpasses the qualification threshold (60) and exceeds the average human score of 68.70, affirming the effectiveness and robustness of the framework. It also enabled a smaller Baichuan2-13B to pass the examination, showcasing the great potential in low-resource settings. Discussion and Conclusion This study shed light on the optimal practices to enhance the capabilities of LLMs in non-English medical scenarios. By synergizing medical knowledge through in-context learning, LLMs can extend clinical insight beyond language barriers in healthcare, significantly reducing language-related disparities of LLM applications and ensuring global benefit in this field.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Knowledge BasesLanguageHumansChinaLicensure, MedicalNatural Language Processing

Stichwörter der Autor:innen

clinical knowledgelarge language modelsmedical examinationnatural language processing

Autor:innen und Institutionen

  • School of Public Health, Zhejiang University School of Medicine, Hangzhou, 310058, China.
  • Jarvis Research Center, Tencent YouTu Lab, Beijing, 100101, China.
  • Department of Nutrition, Harvard T.H. Chan School of Public Health, Boston, MA 02115, United States.
  • Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, MA 02115, United States.

Zitieren

APA

Wu, J., Wu, X., Qiu, Z., Li, M., Lin, S., Zhang, Y., Zheng, Y., Yuan, C., & Yang, J. (2024). Large language models leverage external knowledge to extend clinical insight beyond language boundaries. Journal of the American Medical Informatics Association, 31(9), 2054-2064. https://doi.org/10.1093/jamia/ocae079

Vancouver

Wu J, Wu X, Qiu Z, Li M, Lin S, Zhang Y, et al. Large language models leverage external knowledge to extend clinical insight beyond language boundaries. Journal of the American Medical Informatics Association. 2024;31(9):2054-2064. doi:10.1093/jamia/ocae079

BibTeX
@article{Wu2024,
  author = {Wu, Jiageng and Wu, Xian and Qiu, Zhaopeng and Li, Minghui and Lin, Shixu and Zhang, Yingying and Zheng, Yefeng and Yuan, Changzheng and Yang, Jie},
  title = {Large language models leverage external knowledge to extend clinical insight beyond language boundaries},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  volume = {31},
  number = {9},
  pages = {2054--2064},
  doi = {10.1093/jamia/ocae079}
}
Kostenlos starten →