EUVIMEDEUVIMED entdecken
Crossref

Reality vs Counterfactual: Multi-World Contrastive Reinforcement Learning for Enhancing MLLM’s Theory of Mind in Egocentric Videos

Guiyang Hou, Yihui Fu, Chen Wu, Xiang Huang, Zhe Zheng, Wenqi Zhang, Yongliang Shen, Weiming Lu

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (3) · 1828-1836

DOI: 10.1609/aaai.v40i3.37162 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Guiyang Hou und 7 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Theory of Mind (ToM) refers to the ability to infer others' mental states, which is an essential capability for embodied AI agents to effectively collaborate and interact with humans. While improving Large Language Models' ability to reason about characters' mental states in text-based stories/dialogues has been extensively studied, enhancing Multimodal Large Language Models' ToM capabilities, particularly in egocentric video from an embodied perspective, remains unexplored. In this paper, we propose a contrastive Reinforcement Learning (RL) paradigm that explicitly encourages models to leverage temporal and causal evolutionary patterns in user action sequences to infer user's mental states (goals, beliefs, and potential next actions). Evaluation results on in-domain and out-of-domain demonstrate that our method achieves performance improvements of (+30.00%, +2.00%) and (+5.83%, +5.00%) compared to the backbone model and vanilla Group Relative Policy Optimization (GRPO) model, respectively. Additionally, we compare the performance of two post-training paradigms (Supervise Fine-Tuning and RL) and systematically analyze the reasoning trajectories across the base model, vanilla GRPO model, and our proposed method.

Zitieren

APA

Hou, G., Fu, Y., Wu, C., Huang, X., Zheng, Z., Zhang, W., Shen, Y., & Lu, W. (2026). Reality vs Counterfactual: Multi-World Contrastive Reinforcement Learning for Enhancing MLLM’s Theory of Mind in Egocentric Videos. Proceedings of the AAAI Conference on Artificial Intelligence, 40(3), 1828-1836. https://doi.org/10.1609/aaai.v40i3.37162

Vancouver

Hou G, Fu Y, Wu C, Huang X, Zheng Z, Zhang W, et al. Reality vs Counterfactual: Multi-World Contrastive Reinforcement Learning for Enhancing MLLM’s Theory of Mind in Egocentric Videos. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(3):1828-1836. doi:10.1609/aaai.v40i3.37162

BibTeX
@article{Hou2026,
  author = {Hou, Guiyang and Fu, Yihui and Wu, Chen and Huang, Xiang and Zheng, Zhe and Zhang, Wenqi and Shen, Yongliang and Lu, Weiming},
  title = {Reality vs Counterfactual: Multi-World Contrastive Reinforcement Learning for Enhancing MLLM’s Theory of Mind in Egocentric Videos},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {3},
  pages = {1828--1836},
  doi = {10.1609/aaai.v40i3.37162}
}
Kostenlos starten →