When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (14) · 11955-11963
DOI: 10.1609/aaai.v40i14.38183 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Qilang Ye und 6 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).
Abstract (Originalsprache)
Can Multimodal Large Language Models (MLLMs) discern confused objects that are visually present but audio-absent? To study this, we introduce a new benchmark, AV-ConfuseBench, which simulates an “Audio-Visual Confusion” scene by modifying the corresponding sound of an object in the video, e.g., mute the sounding object and ask MLLMs “Is there a/an {muted-object} sound”. Experimental results reveal that MLLMs, such as Qwen2.5-Omni and Gemini 2.5, struggle to discriminate non-existent audio due to visually dominated reasoning. Motivated by this observation, we introduce RL-CoMM, a Reinforcement Learning-based Collaborative Multi-MLLM that is built upon the Qwen2.5-Omni foundation. RL-CoMM includes two stages: 1) To alleviate visually dominated ambiguities, we introduce an external model, a Large Audio Language Model (LALM), as the reference model to generate audio-only reasoning. Then, we design a Step-wise Reasoning Reward function that enables MLLMs to self-improve audio-visual reasoning with the audio-only reference. 2) To ensure an accurate answer prediction, we introduce Answer-centered Confidence Optimization to reduce the uncertainty of potential heterogeneous reasoning differences. Extensive experiments on audio-visual question answering and audio-visual hallucination show that RL-CoMM improves accuracy by 10~30% over the baseline model with limited training data.
Zitieren
APA
Ye, Q., Zeng, W., Liu, M., Zhang, J., Hu, Y., Yu, Z., & Zhou, Y. (2026). When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?. Proceedings of the AAAI Conference on Artificial Intelligence, 40(14), 11955-11963. https://doi.org/10.1609/aaai.v40i14.38183
Vancouver
Ye Q, Zeng W, Liu M, Zhang J, Hu Y, Yu Z, et al. When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(14):11955-11963. doi:10.1609/aaai.v40i14.38183
BibTeX
@article{Ye2026,
author = {Ye, Qilang and Zeng, Wei and Liu, Meng and Zhang, Jie and Hu, Yupeng and Yu, Zitong and Zhou, Yu},
title = {When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {14},
pages = {11955--11963},
doi = {10.1609/aaai.v40i14.38183}
}
EUVIMED