Evaluating the ChatGPT family of models for biomedical reasoning and classification
Journal of the American Medical Informatics Association · 03.04.2024 · 31 (4) · 940-948
DOI: 10.1093/jamia/ocad256 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 22.01.2024. Verfasst von Shan Chen und 6 weiteren Autor:innen. Beteiligte Einrichtung: Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA 02115, United States. Hauptthemen laut MEDLINE: Language, Natural Language Processing. Die Arbeit wurde bisher 76-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIGMS NIH HHS, NLM NIH HHS, NIH HHS.
Abstract (Originalsprache)
Objective
Large language models (LLMs) have shown impressive ability in biomedical question-answering, but have not been adequately investigated for more specific biomedical applications. This study investigates ChatGPT family of models (GPT-3.5, GPT-4) in biomedical tasks beyond question-answering. Materials and Methods We evaluated model performance with 11 122 samples for two fundamental tasks in the biomedical domain—classification (n = 8676) and reasoning (n = 2446). The first task involves classifying health advice in scientific literature, while the second task is detecting causal relations in biomedical literature. We used 20% of the dataset for prompt development, including zero- and few-shot settings with and without chain-of-thought (CoT). We then evaluated the best prompts from each setting on the remaining dataset, comparing them to models using simple features (BoW with logistic regression) and fine-tuned BioBERT models.
Results
Fine-tuning BioBERT produced the best classification (F1: 0.800-0.902) and reasoning (F1: 0.851) results. Among LLM approaches, few-shot CoT achieved the best classification (F1: 0.671-0.770) and reasoning (F1: 0.682) results, comparable to the BoW model (F1: 0.602-0.753 and 0.675 for classification and reasoning, respectively). It took 78 h to obtain the best LLM results, compared to 0.078 and 0.008 h for the top-performing BioBERT and BoW models, respectively.
Discussion
The simple BoW model performed similarly to the most complex LLM prompting. Prompt engineering required significant investment.
Conclusion
Despite the excitement around viral ChatGPT, fine-tuning for two fundamental biomedical natural language processing tasks remained the best strategy.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA 02115, United States.
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA 02115, United States.
- Computational Health Informatics Program, Boston Children's Hospital, and Harvard Medical School, Boston, MA 02115, United States.
- Ubiquitous Knowledge Processing Lab (UKP Lab), Technical University of Darmstadt, Darmstadt 64289, Germany.
- Radiology and Nuclear Medicine, GROW & CARIM, Maastricht University, Maastricht 6211 LK, Netherlands.
ORCID-Profile: Shan Chen
Förderung
- NIGMS NIH HHS
- NLM NIH HHS
- NIH HHS
- National Institutes of Health
- Woods Foundation
Zitieren
APA
Chen, S., Li, Y., Lu, S., Van, H., Aerts, H. J. W. L., Savova, G. K., & Bitterman, D. S. (2024). Evaluating the ChatGPT family of models for biomedical reasoning and classification. Journal of the American Medical Informatics Association, 31(4), 940-948. https://doi.org/10.1093/jamia/ocad256
Vancouver
Chen S, Li Y, Lu S, Van H, Aerts HJWL, Savova GK, et al. Evaluating the ChatGPT family of models for biomedical reasoning and classification. Journal of the American Medical Informatics Association. 2024;31(4):940-948. doi:10.1093/jamia/ocad256
BibTeX
@article{Chen2024,
author = {Chen, Shan and Li, Yingya and Lu, Sheng and Van, Hoang and Aerts, Hugo J W L and Savova, Guergana K and Bitterman, Danielle S},
title = {Evaluating the ChatGPT family of models for biomedical reasoning and classification},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
volume = {31},
number = {4},
pages = {940--948},
doi = {10.1093/jamia/ocad256}
}
EUVIMED