KGLMQA: enhancing medical visual question answering with knowledge graphs and LLMs
PeerJ Computer Science · 03.03.2026 · 12 · e3679
DOI: 10.7717/peerj-cs.3679 ↗Kurz gefasst
Fachartikel in der Zeitschrift PeerJ Computer Science, online erschienen am 03.03.2026. Verfasst von Wenhu Wang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 56 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Hunan Provincial Natural Science Foundation of China, Postgraduate Innovative Project of Hunan University of Chinese Medicine, Health Research Project of Hunan Provincial Health Commission of China.
Abstract (Originalsprache)
Medical Visual Question Answering (MedVQA) leverages computer vision and natural language processing techniques to assist in clinical decision-making. However, existing models frequently encounter challenges such as restricted multimodal interaction, insufficient guidance from external medical knowledge, and a lack of rigorous diagnostic logic in their responses. To address these issues, we propose KGLMQA, a novel framework that integrates knowledge graphs with Large Language Models (LLMs). The framework comprises three core components: a high-precision MedVQA classification model utilizing gating mechanisms and multi-stage feature fusion; a Knowledge Graph Retrieval Augmented Generation (KGRAG) module for dynamically retrieving and refining structured medical knowledge; and an LLM that generates professional responses based on structured prompts. Experimental results on the Patient-oriented Visual Question Answering (P-VQA), Visual Question Answering in Radiology (VQA-RAD), and Semantically-Labeled Knowledge-Enhanced (SLAKE) datasets demonstrate that KGLMQA achieves state-of-the-art performance in metrics such as Accuracy and Precision. Notably, it outperforms the fully fine-tuned Large Language-and-Vision Assistant for Biomedicine (LLaVA-Med) model in handling open-ended questions on the VQA-RAD dataset. Further error propagation analysis and comparative evaluation with Generative Pre-trained Transformer 4o (GPT-4o) reveal that KGLMQA not only exhibits strong robustness against upstream visual noise but also surpasses GPT-4o in terms of diagnostic logicality. These findings indicate that integrating visual diagnostic cues and explicit structured knowledge with LLMs significantly enhances the interpretability and clinical application potential of MedVQA systems.
Förderung
- Hunan Provincial Natural Science Foundation of China
- Postgraduate Innovative Project of Hunan University of Chinese Medicine
- Health Research Project of Hunan Provincial Health Commission of China
Zitieren
APA
Wang, W., Liu, H., & Wei, C. (2026). KGLMQA: enhancing medical visual question answering with knowledge graphs and LLMs. PeerJ Computer Science, 12, e3679. https://doi.org/10.7717/peerj-cs.3679
Vancouver
Wang W, Liu H, Wei C. KGLMQA: enhancing medical visual question answering with knowledge graphs and LLMs. PeerJ Computer Science. 2026;12:e3679. doi:10.7717/peerj-cs.3679
BibTeX
@article{Wang2026,
author = {Wang, Wenhu and Liu, Huina and Wei, Changfa},
title = {KGLMQA: enhancing medical visual question answering with knowledge graphs and LLMs},
journal = {PeerJ Computer Science},
year = {2026},
volume = {12},
pages = {e3679},
doi = {10.7717/peerj-cs.3679}
}
EUVIMED