Knowledge-Enhanced Image Captioning with Adaptive Graph-based Multimodal Alignment and LLM
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (18) · 15090-15098
DOI: 10.1609/aaai.v40i18.38532 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Guoyi Li und 7 weiteren Autor:innen.
Abstract (Originalsprache)
Image captioning is crucial for multimodal understanding, bridging visual content and natural language. Despite recent advancements in Large Multimodal Models (LMMs), when faced with unseen entities or scenes in the open world, even when attempting to leverage learned knowledge, models still struggle with vague and inaccurate descriptions, and may even generate knowledge hallucinations. A key reason is that the model fails to effectively integrate knowledge with visual information, limiting its understanding of visual content. Thus, we propose Adaptive Knowledge Graph-guided Multimodal Alignment (AKGMA) for image captioning, which enhances semantic understanding in open-world scenes through visual knowledge reasoning, reducing knowledge hallucinations and improving caption quality. It consist three key components: Entity-guided Knowledge Aligner (EKA), Adaptive Knowledge Graph Construction (AKGC), and Scene-Context Knowledge Adapter (SCKA). EKA connects visual entities to knowledge graphs, providing structured knowledge to a small language model, which interacts with a visual encoder to acquire visual knowledge. AKGC uses reinforcement learning to build image-relevant subgraphs to optimize knowledge prompts and improve knowledge hallucinations. SCKA leverages scene graph annotations to extract visual contextual knowledge and inject it into Large Language Models (LLMs), ensuring the generated descriptions are consistent with the image's details. Additionally, we introduce UniKnowCap, a new image knowledge description dataset spanning various open-world knowledge domains, designed to evaluate the knowledge accuracy and detail consistency of model-generated descriptions. Extensive experiments show our model outperforms baselines across multiple metrics.
Zitieren
APA
Li, G., Hu, D., Li, H., Yao, Z., Mi, W., Liu, Z., Zhang, X., & Lyu, H. (2026). Knowledge-Enhanced Image Captioning with Adaptive Graph-based Multimodal Alignment and LLM. Proceedings of the AAAI Conference on Artificial Intelligence, 40(18), 15090-15098. https://doi.org/10.1609/aaai.v40i18.38532
Vancouver
Li G, Hu D, Li H, Yao Z, Mi W, Liu Z, et al. Knowledge-Enhanced Image Captioning with Adaptive Graph-based Multimodal Alignment and LLM. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(18):15090-15098. doi:10.1609/aaai.v40i18.38532
BibTeX
@article{Li2026,
author = {Li, Guoyi and Hu, Die and Li, Haozhe and Yao, Zhongjiang and Mi, Wei and Liu, Zongzhen and Zhang, Xiaodan and Lyu, Honglei},
title = {Knowledge-Enhanced Image Captioning with Adaptive Graph-based Multimodal Alignment and LLM},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {18},
pages = {15090--15098},
doi = {10.1609/aaai.v40i18.38532}
}
EUVIMED