Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (42) · 36021-36029
DOI: 10.1609/aaai.v40i42.40918 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Liu Yu und 6 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref).
Abstract (Originalsprache)
Object hallucination remains a critical challenge in Large Vision-Language Models (LVLMs), where models generate content inconsistent with visual inputs. Existing language-decoder based mitigation approaches often regulate visual or textual attention independently, overlooking their interaction as two key causal factors. To address this, we propose Owl (Bi-mOdal attention reWeighting for Layer-wise hallucination mitigation), a causally-grounded framework that models hallucination process via a structural causal graph, treating decomposed visual and textual attentions as mediators. We introduce VTACR (Visual-to-Textual Attention Contribution Ratio), a novel metric that quantifies the modality contribution imbalance during decoding. Our analysis reveals that hallucinations frequently occur in low-VTACR scenarios, where textual priors dominate and visual grounding is weakened. To mitigate this, we design a fine-grained attention intervention mechanism that dynamically adjusts token- and layer-wise attention guided by VTACR signals. Finally, we propose a dual-path contrastive decoding strategy: one path emphasizes visually grounded predictions, while the other amplifies hallucinated ones -- letting visual truth shine and hallucination collapse. Experimental results on the POPE and CHAIR benchmarks show that Owl achieves significant hallucination reduction, setting a new SOTA in faithfulness while preserving vision-language understanding capability. Our code is available at https://github.com/CikZ2023/OWL
Zitieren
APA
Yu, L., Chen, Z., Kuang, P., Feng, Z., Zhou, F., Wang, L., & Dobbie, G. (2026). Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs. Proceedings of the AAAI Conference on Artificial Intelligence, 40(42), 36021-36029. https://doi.org/10.1609/aaai.v40i42.40918
Vancouver
Yu L, Chen Z, Kuang P, Feng Z, Zhou F, Wang L, et al. Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(42):36021-36029. doi:10.1609/aaai.v40i42.40918
BibTeX
@article{Yu2026,
author = {Yu, Liu and Chen, Zhonghao and Kuang, Ping and Feng, Zhikun and Zhou, Fan and Wang, Lan and Dobbie, Gillian},
title = {Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {42},
pages = {36021--36029},
doi = {10.1609/aaai.v40i42.40918}
}
EUVIMED