EUVIMEDEUVIMED entdecken
Open Access

Evaluating the ChatGPT family of models for biomedical reasoning and classification

Shan Chen iD, Yingya Li, Sheng Lu, Hoang Van, Hugo J W L Aerts, Guergana K Savova, Danielle S Bitterman

Journal of the American Medical Informatics Association · 03.04.2024 · 31 (4) · 940-948

DOI: 10.1093/jamia/ocad256 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 22.01.2024. Verfasst von Shan Chen und 6 weiteren Autor:innen. Beteiligte Einrichtung: Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA 02115, United States. Hauptthemen laut MEDLINE: Language, Natural Language Processing. Die Arbeit wurde bisher 76-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIGMS NIH HHS, NLM NIH HHS, NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Large language models (LLMs) have shown impressive ability in biomedical question-answering, but have not been adequately investigated for more specific biomedical applications. This study investigates ChatGPT family of models (GPT-3.5, GPT-4) in biomedical tasks beyond question-answering. Materials and Methods We evaluated model performance with 11 122 samples for two fundamental tasks in the biomedical domain—classification (n = 8676) and reasoning (n = 2446). The first task involves classifying health advice in scientific literature, while the second task is detecting causal relations in biomedical literature. We used 20% of the dataset for prompt development, including zero- and few-shot settings with and without chain-of-thought (CoT). We then evaluated the best prompts from each setting on the remaining dataset, comparing them to models using simple features (BoW with logistic regression) and fine-tuned BioBERT models.

Results

Fine-tuning BioBERT produced the best classification (F1: 0.800-0.902) and reasoning (F1: 0.851) results. Among LLM approaches, few-shot CoT achieved the best classification (F1: 0.671-0.770) and reasoning (F1: 0.682) results, comparable to the BoW model (F1: 0.602-0.753 and 0.675 for classification and reasoning, respectively). It took 78 h to obtain the best LLM results, compared to 0.078 and 0.008 h for the top-performing BioBERT and BoW models, respectively.

Discussion

The simple BoW model performed similarly to the most complex LLM prompting. Prompt engineering required significant investment.

Conclusion

Despite the excitement around viral ChatGPT, fine-tuning for two fundamental biomedical natural language processing tasks remained the best strategy.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

LanguageNatural Language Processing

Stichwörter der Autor:innen

ChatGPTbiomedical researchclassificationnatural language processingreasoning

Autor:innen und Institutionen

  • Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA 02115, United States.
  • Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA 02115, United States.
  • Computational Health Informatics Program, Boston Children's Hospital, and Harvard Medical School, Boston, MA 02115, United States.
  • Ubiquitous Knowledge Processing Lab (UKP Lab), Technical University of Darmstadt, Darmstadt 64289, Germany.
  • Radiology and Nuclear Medicine, GROW & CARIM, Maastricht University, Maastricht 6211 LK, Netherlands.

ORCID-Profile: Shan Chen

Förderung

  • NIGMS NIH HHS
  • NLM NIH HHS
  • NIH HHS
  • National Institutes of Health
  • Woods Foundation

Zitieren

APA

Chen, S., Li, Y., Lu, S., Van, H., Aerts, H. J. W. L., Savova, G. K., & Bitterman, D. S. (2024). Evaluating the ChatGPT family of models for biomedical reasoning and classification. Journal of the American Medical Informatics Association, 31(4), 940-948. https://doi.org/10.1093/jamia/ocad256

Vancouver

Chen S, Li Y, Lu S, Van H, Aerts HJWL, Savova GK, et al. Evaluating the ChatGPT family of models for biomedical reasoning and classification. Journal of the American Medical Informatics Association. 2024;31(4):940-948. doi:10.1093/jamia/ocad256

BibTeX
@article{Chen2024,
  author = {Chen, Shan and Li, Yingya and Lu, Sheng and Van, Hoang and Aerts, Hugo J W L and Savova, Guergana K and Bitterman, Danielle S},
  title = {Evaluating the ChatGPT family of models for biomedical reasoning and classification},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  volume = {31},
  number = {4},
  pages = {940--948},
  doi = {10.1093/jamia/ocad256}
}
Kostenlos starten →