Improving large language models for clinical named entity recognition via prompt engineering
Journal of the American Medical Informatics Association · 01.09.2024 · 31 (9) · 1812-1820
DOI: 10.1093/jamia/ocad259 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 27.01.2024. Verfasst von Yan Hu und 11 weiteren Autor:innen. Beteiligte Einrichtung: McWilliams School of Biomedical Informatics, Houston, TX, United States. Hauptthemen laut MEDLINE: Natural Language Processing, Electronic Health Records. Die Arbeit wurde bisher 336-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIA NIH HHS, NIBIB NIH HHS, NCATS NIH HHS.
Abstract (Originalsprache)
Importance The study highlights the potential of large language models, specifically GPT-3.5 and GPT-4, in processing complex clinical data and extracting meaningful information with minimal training data. By developing and refining prompt-based strategies, we can significantly enhance the models’ performance, making them viable tools for clinical NER tasks and possibly reducing the reliance on extensive annotated datasets.
Objectives
This study quantifies the capabilities of GPT-3.5 and GPT-4 for clinical named entity recognition (NER) tasks and proposes task-specific prompts to improve their performance. Materials and Methods We evaluated these models on 2 clinical NER tasks: (1) to extract medical problems, treatments, and tests from clinical notes in the MTSamples corpus, following the 2010 i2b2 concept extraction shared task, and (2) to identify nervous system disorder-related adverse events from safety reports in the vaccine adverse event reporting system (VAERS). To improve the GPT models' performance, we developed a clinical task-specific prompt framework that includes (1) baseline prompts with task description and format specification, (2) annotation guideline-based prompts, (3) error analysis-based instructions, and (4) annotated samples for few-shot learning. We assessed each prompt's effectiveness and compared the models to BioClinicalBERT.
Results
Using baseline prompts, GPT-3.5 and GPT-4 achieved relaxed F1 scores of 0.634, 0.804 for MTSamples and 0.301, 0.593 for VAERS. Additional prompt components consistently improved model performance. When all 4 components were used, GPT-3.5 and GPT-4 achieved relaxed F1 socres of 0.794, 0.861 for MTSamples and 0.676, 0.736 for VAERS, demonstrating the effectiveness of our prompt framework. Although these results trail BioClinicalBERT (F1 of 0.901 for the MTSamples dataset and 0.802 for the VAERS), it is very promising considering few training samples are needed.
Discussion
The study’s findings suggest a promising direction in leveraging LLMs for clinical NER tasks. However, while the performance of GPT models improved with task-specific prompts, there's a need for further development and refinement. LLMs like GPT-4 show potential in achieving close performance to state-of-the-art models like BioClinicalBERT, but they still require careful prompt engineering and understanding of task-specific knowledge. The study also underscores the importance of evaluation schemas that accurately reflect the capabilities and performance of LLMs in clinical settings.
Conclusion
While direct application of GPT models to clinical NER tasks falls short of optimal performance, our task-specific prompt framework, incorporating medical knowledge and training samples, significantly enhances GPT models' feasibility for potential clinical applications.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- McWilliams School of Biomedical Informatics, Houston, TX, United States.
- Section of Biomedical Informatics and Data Science, School of Medicine, Yale University, New Haven, CT, United States.
- National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, MD, United States.
ORCID-Profile: Yan Hu, Jingcheng Du, Yujia Zhou, Xiaoqian Jiang
Förderung
- NIA NIH HHS
- NIBIB NIH HHS
- NCATS NIH HHS
- NLM NIH HHS
- NIH HHS
- NIAID NIH HHS
- Cancer Prevention and Research Institute of Texas
- National Science Foundation
Zitieren
APA
Hu, Y., Chen, Q., Du, J., Peng, X., Keloth, V. K., Zuo, X., Zhou, Y., Li, Z., Jiang, X., Lu, Z., Roberts, K., & Xu, H. (2024). Improving large language models for clinical named entity recognition via prompt engineering. Journal of the American Medical Informatics Association, 31(9), 1812-1820. https://doi.org/10.1093/jamia/ocad259
Vancouver
Hu Y, Chen Q, Du J, Peng X, Keloth VK, Zuo X, et al. Improving large language models for clinical named entity recognition via prompt engineering. Journal of the American Medical Informatics Association. 2024;31(9):1812-1820. doi:10.1093/jamia/ocad259
BibTeX
@article{Hu2024,
author = {Hu, Yan and Chen, Qingyu and Du, Jingcheng and Peng, Xueqing and Keloth, Vipina Kuttichi and Zuo, Xu and Zhou, Yujia and Li, Zehan and Jiang, Xiaoqian and Lu, Zhiyong and Roberts, Kirk and Xu, Hua},
title = {Improving large language models for clinical named entity recognition via prompt engineering},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
volume = {31},
number = {9},
pages = {1812--1820},
doi = {10.1093/jamia/ocad259}
}
EUVIMED