Reality vs Counterfactual: Multi-World Contrastive Reinforcement Learning for Enhancing MLLM’s Theory of Mind in Egocentric Videos
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (3) · 1828-1836
DOI: 10.1609/aaai.v40i3.37162 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Guiyang Hou und 7 weiteren Autor:innen.
Abstract (Originalsprache)
Theory of Mind (ToM) refers to the ability to infer others' mental states, which is an essential capability for embodied AI agents to effectively collaborate and interact with humans. While improving Large Language Models' ability to reason about characters' mental states in text-based stories/dialogues has been extensively studied, enhancing Multimodal Large Language Models' ToM capabilities, particularly in egocentric video from an embodied perspective, remains unexplored. In this paper, we propose a contrastive Reinforcement Learning (RL) paradigm that explicitly encourages models to leverage temporal and causal evolutionary patterns in user action sequences to infer user's mental states (goals, beliefs, and potential next actions). Evaluation results on in-domain and out-of-domain demonstrate that our method achieves performance improvements of (+30.00%, +2.00%) and (+5.83%, +5.00%) compared to the backbone model and vanilla Group Relative Policy Optimization (GRPO) model, respectively. Additionally, we compare the performance of two post-training paradigms (Supervise Fine-Tuning and RL) and systematically analyze the reasoning trajectories across the base model, vanilla GRPO model, and our proposed method.
Zitieren
APA
Hou, G., Fu, Y., Wu, C., Huang, X., Zheng, Z., Zhang, W., Shen, Y., & Lu, W. (2026). Reality vs Counterfactual: Multi-World Contrastive Reinforcement Learning for Enhancing MLLM’s Theory of Mind in Egocentric Videos. Proceedings of the AAAI Conference on Artificial Intelligence, 40(3), 1828-1836. https://doi.org/10.1609/aaai.v40i3.37162
Vancouver
Hou G, Fu Y, Wu C, Huang X, Zheng Z, Zhang W, et al. Reality vs Counterfactual: Multi-World Contrastive Reinforcement Learning for Enhancing MLLM’s Theory of Mind in Egocentric Videos. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(3):1828-1836. doi:10.1609/aaai.v40i3.37162
BibTeX
@article{Hou2026,
author = {Hou, Guiyang and Fu, Yihui and Wu, Chen and Huang, Xiang and Zheng, Zhe and Zhang, Wenqi and Shen, Yongliang and Lu, Weiming},
title = {Reality vs Counterfactual: Multi-World Contrastive Reinforcement Learning for Enhancing MLLM’s Theory of Mind in Egocentric Videos},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {3},
pages = {1828--1836},
doi = {10.1609/aaai.v40i3.37162}
}
EUVIMED