EUVIMEDEUVIMED entdecken
Crossref

When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?

Qilang Ye, Wei Zeng, Meng Liu, Jie Zhang, Yupeng Hu, Zitong Yu, Yu Zhou

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (14) · 11955-11963

DOI: 10.1609/aaai.v40i14.38183 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Qilang Ye und 6 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Can Multimodal Large Language Models (MLLMs) discern confused objects that are visually present but audio-absent? To study this, we introduce a new benchmark, AV-ConfuseBench, which simulates an “Audio-Visual Confusion” scene by modifying the corresponding sound of an object in the video, e.g., mute the sounding object and ask MLLMs “Is there a/an {muted-object} sound”. Experimental results reveal that MLLMs, such as Qwen2.5-Omni and Gemini 2.5, struggle to discriminate non-existent audio due to visually dominated reasoning. Motivated by this observation, we introduce RL-CoMM, a Reinforcement Learning-based Collaborative Multi-MLLM that is built upon the Qwen2.5-Omni foundation. RL-CoMM includes two stages: 1) To alleviate visually dominated ambiguities, we introduce an external model, a Large Audio Language Model (LALM), as the reference model to generate audio-only reasoning. Then, we design a Step-wise Reasoning Reward function that enables MLLMs to self-improve audio-visual reasoning with the audio-only reference. 2) To ensure an accurate answer prediction, we introduce Answer-centered Confidence Optimization to reduce the uncertainty of potential heterogeneous reasoning differences. Extensive experiments on audio-visual question answering and audio-visual hallucination show that RL-CoMM improves accuracy by 10~30% over the baseline model with limited training data.

Zitieren

APA

Ye, Q., Zeng, W., Liu, M., Zhang, J., Hu, Y., Yu, Z., & Zhou, Y. (2026). When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?. Proceedings of the AAAI Conference on Artificial Intelligence, 40(14), 11955-11963. https://doi.org/10.1609/aaai.v40i14.38183

Vancouver

Ye Q, Zeng W, Liu M, Zhang J, Hu Y, Yu Z, et al. When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(14):11955-11963. doi:10.1609/aaai.v40i14.38183

BibTeX
@article{Ye2026,
  author = {Ye, Qilang and Zeng, Wei and Liu, Meng and Zhang, Jie and Hu, Yupeng and Yu, Zitong and Zhou, Yu},
  title = {When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {14},
  pages = {11955--11963},
  doi = {10.1609/aaai.v40i14.38183}
}
Kostenlos starten →