AnaToM: A Dataset Generation Framework for Evaluating Theory of Mind Reasoning Toward Anatomy of Difficulty Through Structurally Controlled Story Generation
Journal of Natural Language Processing · 2026 · 33 (2) · 630-657
DOI: 10.5715/jnlp.33.630 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Natural Language Processing, 2026. Verfasst von Jundai Suzuki und 2 weiteren Autor:innen. Die Arbeit nennt 32 Literaturangaben.
Abstract (Originalsprache)
Evaluating Theory of Mind (ToM) in Large Language Models (LLMs) is an important area of research for understanding the social intelligence of artificial intelligence. Recent ToM benchmarks have significantly enhanced the complexity, comprehensiveness, and practicality of evaluations.However, while the focus has been on constructing “more difficult” or “more comprehensive” tasks, systematic analysis of structural factors that inherently determine the difficulty of ToM reasoning, i.e., “what” makes reasoning difficult, is insufficient.Hence, we propose a new dataset generation framework for ToM evaluation, named AnaToM.To realize an “anatomy of difficulty” in ToM reasoning, AnaToM strictly controls structural parameters such as the number of entities and the timeline in a story.This parameter control enables the isolation and identification of factors affecting ToM in LLMs, thereby enabling a more precise examination of their reasoning mechanisms.The proposed framework provides a systematic methodology for diagnosing the structural limits of LLM reasoning abilities, thus offering a foundational diagnostic baseline that complements the evaluation of broader sociocognitive capabilities in future benchmark designs.
Zitieren
APA
Suzuki, J., Ishigaki, R., & Maeda, E. (2026). AnaToM: A Dataset Generation Framework for Evaluating Theory of Mind Reasoning Toward Anatomy of Difficulty Through Structurally Controlled Story Generation. Journal of Natural Language Processing, 33(2), 630-657. https://doi.org/10.5715/jnlp.33.630
Vancouver
Suzuki J, Ishigaki R, Maeda E. AnaToM: A Dataset Generation Framework for Evaluating Theory of Mind Reasoning Toward Anatomy of Difficulty Through Structurally Controlled Story Generation. Journal of Natural Language Processing. 2026;33(2):630-657. doi:10.5715/jnlp.33.630
BibTeX
@article{Suzuki2026,
author = {Suzuki, Jundai and Ishigaki, Ryoma and Maeda, Eisaku},
title = {AnaToM: A Dataset Generation Framework for Evaluating Theory of Mind Reasoning Toward Anatomy of Difficulty Through Structurally Controlled Story Generation},
journal = {Journal of Natural Language Processing},
year = {2026},
volume = {33},
number = {2},
pages = {630--657},
doi = {10.5715/jnlp.33.630}
}
EUVIMED