EUVIMEDEUVIMED entdecken
Crossref

AnaToM: A Dataset Generation Framework for Evaluating Theory of Mind Reasoning Toward Anatomy of Difficulty Through Structurally Controlled Story Generation

Jundai Suzuki, Ryoma Ishigaki, Eisaku Maeda

Journal of Natural Language Processing · 2026 · 33 (2) · 630-657

DOI: 10.5715/jnlp.33.630 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Natural Language Processing, 2026. Verfasst von Jundai Suzuki und 2 weiteren Autor:innen. Die Arbeit nennt 32 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Evaluating Theory of Mind (ToM) in Large Language Models (LLMs) is an important area of research for understanding the social intelligence of artificial intelligence. Recent ToM benchmarks have significantly enhanced the complexity, comprehensiveness, and practicality of evaluations.However, while the focus has been on constructing “more difficult” or “more comprehensive” tasks, systematic analysis of structural factors that inherently determine the difficulty of ToM reasoning, i.e., “what” makes reasoning difficult, is insufficient.Hence, we propose a new dataset generation framework for ToM evaluation, named AnaToM.To realize an “anatomy of difficulty” in ToM reasoning, AnaToM strictly controls structural parameters such as the number of entities and the timeline in a story.This parameter control enables the isolation and identification of factors affecting ToM in LLMs, thereby enabling a more precise examination of their reasoning mechanisms.The proposed framework provides a systematic methodology for diagnosing the structural limits of LLM reasoning abilities, thus offering a foundational diagnostic baseline that complements the evaluation of broader sociocognitive capabilities in future benchmark designs.

Zitieren

APA

Suzuki, J., Ishigaki, R., & Maeda, E. (2026). AnaToM: A Dataset Generation Framework for Evaluating Theory of Mind Reasoning Toward Anatomy of Difficulty Through Structurally Controlled Story Generation. Journal of Natural Language Processing, 33(2), 630-657. https://doi.org/10.5715/jnlp.33.630

Vancouver

Suzuki J, Ishigaki R, Maeda E. AnaToM: A Dataset Generation Framework for Evaluating Theory of Mind Reasoning Toward Anatomy of Difficulty Through Structurally Controlled Story Generation. Journal of Natural Language Processing. 2026;33(2):630-657. doi:10.5715/jnlp.33.630

BibTeX
@article{Suzuki2026,
  author = {Suzuki, Jundai and Ishigaki, Ryoma and Maeda, Eisaku},
  title = {AnaToM: A Dataset Generation Framework for Evaluating Theory of Mind Reasoning Toward Anatomy of Difficulty Through Structurally Controlled Story Generation},
  journal = {Journal of Natural Language Processing},
  year = {2026},
  volume = {33},
  number = {2},
  pages = {630--657},
  doi = {10.5715/jnlp.33.630}
}
Kostenlos starten →