An extensive benchmark study on biomedical text generation and mining with ChatGPT
Bioinformatics · 2023
DOI: 10.1093/bioinformatics/btad557 ↗Kurz gefasst
Fachartikel in der Zeitschrift Bioinformatics, 2023. Verfasst von Qijie Chen und 9 weiteren Autor:innen. Beteiligte Einrichtung: AIDD, Mindrank AI Ltd, Zhejiang 310000, China. Hauptthemen laut MEDLINE: Natural Language Processing, Data Mining. Die Arbeit wurde bisher 95-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Motivation In recent years, the development of natural language process (NLP) technologies and deep learning hardware has led to significant improvement in large language models (LLMs). The ChatGPT, the state-of-the-art LLM built on GPT-3.5 and GPT-4, shows excellent capabilities in general language understanding and reasoning. Researchers also tested the GPTs on a variety of NLP-related tasks and benchmarks and got excellent results. With exciting performance on daily chat, researchers began to explore the capacity of ChatGPT on expertise that requires professional education for human and we are interested in the biomedical domain. Results To evaluate the performance of ChatGPT on biomedical-related tasks, this article presents a comprehensive benchmark study on the use of ChatGPT for biomedical corpus, including article abstracts, clinical trials description, biomedical questions, and so on. Typical NLP tasks like named entity recognization, relation extraction, sentence similarity, question and answering, and document classification are included. Overall, ChatGPT got a BLURB score of 58.50 while the state-of-the-art model had a score of 84.30. Through a series of experiments, we demonstrated the effectiveness and versatility of ChatGPT in biomedical text understanding, reasoning and generation, and the limitation of ChatGPT build on GPT-3.5. Availability and implementation All the datasets are available from BLURB benchmark https://microsoft.github.io/BLURB/index.html. The prompts are described in the article.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- AIDD, Mindrank AI Ltd, Zhejiang 310000, China.
- College of Life Sciences, Nankai University, Tianjin 300071, China.
- Guangzhou Laboratory, GuangDong 510005, China.
- National Heart and Lung Institute, Imperial College London, London, United Kingdom.
ORCID-Profile: Qijie Chen, Zhangmin Niu
Zitieren
APA
Chen, Q., Sun, H., Liu, H., Jiang, Y., Ran, T., Jin, X., Xiao, X., Lin, Z., Chen, H., & Niu, Z. (2023). An extensive benchmark study on biomedical text generation and mining with ChatGPT. Bioinformatics. https://doi.org/10.1093/bioinformatics/btad557
Vancouver
Chen Q, Sun H, Liu H, Jiang Y, Ran T, Jin X, et al. An extensive benchmark study on biomedical text generation and mining with ChatGPT. Bioinformatics. 2023. doi:10.1093/bioinformatics/btad557
BibTeX
@article{Chen2023,
author = {Chen, Qijie and Sun, Haotong and Liu, Haoyang and Jiang, Yinghui and Ran, Ting and Jin, Xurui and Xiao, Xianglu and Lin, Zhimin and Chen, Hongming and Niu, Zhangmin},
title = {An extensive benchmark study on biomedical text generation and mining with ChatGPT},
journal = {Bioinformatics},
year = {2023},
doi = {10.1093/bioinformatics/btad557}
}
EUVIMED