An Automatic and End-to-End System for Rare Disease Knowledge Graph Construction Based on Ontology-Enhanced Large Language Models: Development Study
JMIR Medical Informatics · 18.12.2024 · 12 · e60665-e60665
DOI: 10.2196/60665 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Medical Informatics, online erschienen am 18.12.2024. Verfasst von Lang Cao und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, University of Illinois Urbana-Champaign, Urbana, IL, United States. Hauptthemen laut MEDLINE: Rare Diseases, Data Mining, Biological Ontologies. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Rare diseases affect millions worldwide but sometimes face limited research focus individually due to low prevalence. Many rare diseases do not have specific International Classification of Diseases, Ninth Edition ( ICD-9 ) and Tenth Edition ( ICD-10 ), codes and therefore cannot be reliably extracted from granular fields like “Diagnosis” and “Problem List” entries, which complicates tasks that require identification of patients with these conditions, including clinical trial recruitment and research efforts. Recent advancements in large language models (LLMs) have shown promise in automating the extraction of medical information, offering the potential to improve medical research, diagnosis, and management. However, most LLMs lack professional medical knowledge, especially concerning specific rare diseases, and cannot effectively manage rare disease data in its various ontological forms, making it unsuitable for these tasks.
Objective
Our aim is to create an end-to-end system called automated rare disease mining (AutoRD), which automates the extraction of rare disease–related information from medical text, focusing on entities and their relations to other medical concepts, such as signs and symptoms. AutoRD integrates up-to-date ontologies with other structured knowledge and demonstrates superior performance in rare disease extraction tasks. We conducted various experiments to evaluate AutoRD’s performance, aiming to surpass common LLMs and traditional methods.
Methods
AutoRD is a pipeline system that involves data preprocessing, entity extraction, relation extraction, entity calibration, and knowledge graph construction. We implemented this system using GPT-4 and medical knowledge graphs developed from the open-source Human Phenotype and Orphanet ontologies, using techniques such as chain-of-thought reasoning and prompt engineering. We quantitatively evaluated our system’s performance in entity extraction, relation extraction, and knowledge graph construction. The experiment used the well-curated dataset RareDis2023, which contains medical literature focused on rare disease entities and their relations, making it an ideal dataset for training and testing our methodology.
Results
On the RareDis2023 dataset, AutoRD achieved an overall entity extraction F 1 -score of 56.1% and a relation extraction F 1 -score of 38.6%, marking a 14.4% improvement over the baseline LLM. Notably, the F 1 -score for rare disease entity extraction reached 83.5%, indicating high precision and recall in identifying rare disease mentions. These results demonstrate the effectiveness of integrating LLMs with medical ontologies in extracting complex rare disease information.
Conclusions
AutoRD is an automated end-to-end system for extracting rare disease information from text to build knowledge graphs, addressing critical limitations of existing LLMs by improving identification of these diseases and connecting them to related clinical features. This work underscores the significant potential of LLMs in transforming health care, particularly in the rare disease domain. By leveraging ontology-enhanced LLMs, AutoRD constructs a robust medical knowledge base that incorporates up-to-date rare disease information, facilitating improved identification of patients and resulting in more inclusive research and trial candidacy efforts.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science, University of Illinois Urbana-Champaign, Urbana, IL, United States.
- Department of Pediatrics, University of Illinois College of Medicine Peoria, Peoria, IL, United States.
ORCID-Profile: Lang Cao, Jimeng Sun, Adam Cross
Zitieren
APA
Cao, L., Sun, J., & Cross, A. (2024). An Automatic and End-to-End System for Rare Disease Knowledge Graph Construction Based on Ontology-Enhanced Large Language Models: Development Study. JMIR Medical Informatics, 12, e60665-e60665. https://doi.org/10.2196/60665
Vancouver
Cao L, Sun J, Cross A. An Automatic and End-to-End System for Rare Disease Knowledge Graph Construction Based on Ontology-Enhanced Large Language Models: Development Study. JMIR Medical Informatics. 2024;12:e60665-e60665. doi:10.2196/60665
BibTeX
@article{Cao2024,
author = {Cao, Lang and Sun, Jimeng and Cross, Adam},
title = {An Automatic and End-to-End System for Rare Disease Knowledge Graph Construction Based on Ontology-Enhanced Large Language Models: Development Study},
journal = {JMIR Medical Informatics},
year = {2024},
volume = {12},
pages = {e60665--e60665},
doi = {10.2196/60665}
}
EUVIMED