Prompting large language models to extract chemical‒disease relation precisely and comprehensively at the document level: an evaluation study
PLOS ONE · 2025
DOI: 10.1371/journal.pone.0320123 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS ONE, 2025. Verfasst von Mei Chen und 2 weiteren Autor:innen. Beteiligte Einrichtung: Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China, Beijing 100081, China. Hauptthemen laut MEDLINE: Data Mining, Natural Language Processing. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Given the scarcity of annotated data, current deep learning methods face challenges in the field of document-level chemical-disease relation extraction, making it difficult to achieve precise relation extraction capable of identifying relation types and comprehensive extraction tasks that identify relation-related factors. This study tests the abilities of three large language models (LLMs), GPT3.5, GPT4.0, and Claude-opus, to perform precise and comprehensive extraction in document-level chemical-disease relation extraction on a self-constructed dataset. Firstly, based on the task characteristics, this study designs six workflows for precise extraction and five workflows for comprehensive extraction using prompting engineering strategies. The characteristics of the extraction process are analyzed through the performance differences under different workflows. Secondly, this study analyzes the content bias in LLMs extraction by examining the extraction effectiveness of different workflows on different types of content. Finally, this study analyzes the error characteristics of extracting incorrect examples by the LLMs. The experimental results show that: (1) The LLMs demonstrate good extraction capabilities, achieving the highest F1 scores of 87% and 73% respectively in the tasks of precise extraction and comprehensive extraction; (2) In the extraction process, the LLMs exhibit a certain degree of stubbornness, with limited effectiveness of prompting engineering strategies; (3) In terms of extraction content, the LLMs show a content bias, with stronger abilities to identify positive relations such as induction and acceleration; (4) The essence of extraction errors lies in the LLMs’ misunderstanding of the implicit meanings in biomedical texts. This study provides practical workflows for precise and comprehensive extraction of document-level chemical-disease relations and also indicates that optimizing training data is the key to building more efficient and accurate extraction methods in the future.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China, Beijing 100081, China.
- School of Information Engineering, Minzu University of China, Beijing 100081, China.
ORCID-Profile: Mei Chen
Zitieren
APA
Chen, M., Zhang, T., & Wang, S. (2025). Prompting large language models to extract chemical‒disease relation precisely and comprehensively at the document level: an evaluation study. PLOS ONE. https://doi.org/10.1371/journal.pone.0320123
Vancouver
Chen M, Zhang T, Wang S. Prompting large language models to extract chemical‒disease relation precisely and comprehensively at the document level: an evaluation study. PLOS ONE. 2025. doi:10.1371/journal.pone.0320123
BibTeX
@article{Chen2025,
author = {Chen, Mei and Zhang, Tingting and Wang, Shibin},
title = {Prompting large language models to extract chemical‒disease relation precisely and comprehensively at the document level: an evaluation study},
journal = {PLOS ONE},
year = {2025},
doi = {10.1371/journal.pone.0320123}
}
EUVIMED