EUVIMEDEUVIMED entdecken
Crossref

Enhancing Retrieval-Augmented Large Vision Language Models via Knowledge Conflict Mitigation

Wenbin An, Jiahao Nie, Feng Tian, Mingxiang Cai, Yaqiang Wu, Xiaoqin Zhang, Shijian Lu

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (4) · 2318-2326

DOI: 10.1609/aaai.v40i4.37216 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Wenbin An und 6 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal Retrieval-Augmented Generation (MRAG) has recently been explored to empower Large Vision Language Models (LVLMs) with more comprehensive and up-to-date contextual knowledge, aiming to compensate for their limited and coarse-grained parametric knowledge in knowledge-intensive tasks. However, the retrieved contextual knowledge is usually not aligned with LVLMs’ internal parametric knowledge, leading to knowledge conflicts and further unreliable responses. To tackle this issue, we design KCM, a training-free and plug-and-play framework that can effectively mitigate knowledge conflicts while incorporating MRAG for more accurate LVLM responses. KCM enhances contextual knowledge utilization by modifying the LVLM architecture from three key perspectives. First, KCM adaptively adjusts attention distributions among multiple attention heads, encouraging LVLMs to focus on contextual knowledge with reduced distraction. Second, KCM identifies and prunes knowledge-centric LVLM neurons that encode coarse-grained parametric knowledge, thereby suppressing interferences and enabling more effective integration of contextual knowledge. Third, KCM amplifies the information flow from the input context by injecting supplementary context logits, reinforcing its contribution to the final output. Extensive experiments over multiple LVLMs and benchmarks show that KCM outperforms the state-of-the-art consistently by large margins, incurring neither extra training nor external tools.

Zitieren

APA

An, W., Nie, J., Tian, F., Cai, M., Wu, Y., Zhang, X., & Lu, S. (2026). Enhancing Retrieval-Augmented Large Vision Language Models via Knowledge Conflict Mitigation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(4), 2318-2326. https://doi.org/10.1609/aaai.v40i4.37216

Vancouver

An W, Nie J, Tian F, Cai M, Wu Y, Zhang X, et al. Enhancing Retrieval-Augmented Large Vision Language Models via Knowledge Conflict Mitigation. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(4):2318-2326. doi:10.1609/aaai.v40i4.37216

BibTeX
@article{An2026,
  author = {An, Wenbin and Nie, Jiahao and Tian, Feng and Cai, Mingxiang and Wu, Yaqiang and Zhang, Xiaoqin and Lu, Shijian},
  title = {Enhancing Retrieval-Augmented Large Vision Language Models via Knowledge Conflict Mitigation},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {4},
  pages = {2318--2326},
  doi = {10.1609/aaai.v40i4.37216}
}
Kostenlos starten →