EUVIMEDEUVIMED entdecken
Crossref

MuMA-ToM: Multi-modal Multi-Agent Theory of Mind

Haojun Shi, Suyu Ye, Xinyu Fang, Chuanyang Jin, Leyla Isik, Yen-Ling Kuo, Tianmin Shu

Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (2) · 1510-1519

DOI: 10.1609/aaai.v39i2.32142 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Haojun Shi und 6 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Understanding people's social interactions in complex real-world scenarios often relies on intricate mental reasoning. To truly understand how and why people interact with one another, we must infer the underlying mental states that give rise to the social interactions, i.e., Theory of Mind reasoning in multi-agent interactions. Additionally, social interactions are often multi-modal -- we can watch people's actions, hear their conversations, and/or read about their past behaviors. For AI systems to successfully and safely interact with people in real-world environments, they also need to understand people's mental states as well as their inferences about each other's mental states based on multi-modal information about their interactions. For this, we introduce MuMA-ToM, a Multi-modal Multi-Agent Theory of Mind benchmark. MuMA-ToM is the first multi-modal Theory of Mind benchmark that evaluates mental reasoning in embodied multi-agent interactions. In MuMA-ToM, we provide video and text descriptions of people's multi-modal behavior in realistic household environments. Based on the context, we then ask questions about people's goals, beliefs, and beliefs about others' goals. We validated MuMA-ToM in a human experiment and provided a human baseline. We also proposed a novel multi-modal, multi-agent ToM model, LIMP (Language model-based Inverse Multi-agent Planning). Our experimental results show that LIMP significantly outperforms state-of-the-art methods, including large multi-modal models (e.g., GPT-4o, Gemini-1.5 Pro) and a recent multi-modal ToM model, BIP-ALM.

Zitieren

APA

Shi, H., Ye, S., Fang, X., Jin, C., Isik, L., Kuo, Y., & Shu, T. (2025). MuMA-ToM: Multi-modal Multi-Agent Theory of Mind. Proceedings of the AAAI Conference on Artificial Intelligence, 39(2), 1510-1519. https://doi.org/10.1609/aaai.v39i2.32142

Vancouver

Shi H, Ye S, Fang X, Jin C, Isik L, Kuo Y, et al. MuMA-ToM: Multi-modal Multi-Agent Theory of Mind. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(2):1510-1519. doi:10.1609/aaai.v39i2.32142

BibTeX
@article{Shi2025,
  author = {Shi, Haojun and Ye, Suyu and Fang, Xinyu and Jin, Chuanyang and Isik, Leyla and Kuo, Yen-Ling and Shu, Tianmin},
  title = {MuMA-ToM: Multi-modal Multi-Agent Theory of Mind},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2025},
  volume = {39},
  number = {2},
  pages = {1510--1519},
  doi = {10.1609/aaai.v39i2.32142}
}
Kostenlos starten →