EUVIMEDEUVIMED entdecken
Crossref

Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs

Liu Yu, Zhonghao Chen, Ping Kuang, Zhikun Feng, Fan Zhou, Lan Wang, Gillian Dobbie

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (42) · 36021-36029

DOI: 10.1609/aaai.v40i42.40918 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Liu Yu und 6 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Object hallucination remains a critical challenge in Large Vision-Language Models (LVLMs), where models generate content inconsistent with visual inputs. Existing language-decoder based mitigation approaches often regulate visual or textual attention independently, overlooking their interaction as two key causal factors. To address this, we propose Owl (Bi-mOdal attention reWeighting for Layer-wise hallucination mitigation), a causally-grounded framework that models hallucination process via a structural causal graph, treating decomposed visual and textual attentions as mediators. We introduce VTACR (Visual-to-Textual Attention Contribution Ratio), a novel metric that quantifies the modality contribution imbalance during decoding. Our analysis reveals that hallucinations frequently occur in low-VTACR scenarios, where textual priors dominate and visual grounding is weakened. To mitigate this, we design a fine-grained attention intervention mechanism that dynamically adjusts token- and layer-wise attention guided by VTACR signals. Finally, we propose a dual-path contrastive decoding strategy: one path emphasizes visually grounded predictions, while the other amplifies hallucinated ones -- letting visual truth shine and hallucination collapse. Experimental results on the POPE and CHAIR benchmarks show that Owl achieves significant hallucination reduction, setting a new SOTA in faithfulness while preserving vision-language understanding capability. Our code is available at https://github.com/CikZ2023/OWL

Zitieren

APA

Yu, L., Chen, Z., Kuang, P., Feng, Z., Zhou, F., Wang, L., & Dobbie, G. (2026). Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs. Proceedings of the AAAI Conference on Artificial Intelligence, 40(42), 36021-36029. https://doi.org/10.1609/aaai.v40i42.40918

Vancouver

Yu L, Chen Z, Kuang P, Feng Z, Zhou F, Wang L, et al. Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(42):36021-36029. doi:10.1609/aaai.v40i42.40918

BibTeX
@article{Yu2026,
  author = {Yu, Liu and Chen, Zhonghao and Kuang, Ping and Feng, Zhikun and Zhou, Fan and Wang, Lan and Dobbie, Gillian},
  title = {Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {42},
  pages = {36021--36029},
  doi = {10.1609/aaai.v40i42.40918}
}
Kostenlos starten →