M o N a C o : More Natural and Complex Questions for Reasoning Across Dozens of Documents
Transactions of the Association for Computational Linguistics · 05.01.2026 · 14 · 23-46
DOI: 10.1162/tacl.a.64 ↗Kurz gefasst
Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 05.01.2026. Verfasst von Tomer Wolfson und 7 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 71 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Automated agents, powered by large language models (LLMs), are emerging as the go-to tool for querying information. However, evaluation benchmarks for LLM agents rarely feature natural questions that are both information-seeking and genuinely time-consuming for humans. To address this gap we introduce MoNaCo, a benchmark of 1,315 natural and time-consuming questions that require dozens, and at times hundreds, of intermediate steps to solve— far more than any existing QA benchmark. To build MoNaCo, we developed a decomposed annotation pipeline to elicit and manually answer real-world time-consuming questions at scale. Frontier LLMs evaluated on MoNaCo achieve at most 61.2% F1, hampered by low recall and hallucinations. Our results underscore the limitations of LLM-powered agents in handling the complexity and sheer breadth of real-world information-seeking tasks—with MoNaCo providing an effective resource for tracking such progress. The MoNaCo benchmark, codebase, prompts, and models predictions are all publicly available at: https://tomerwolgithub.github.io/monaco.
Zitieren
APA
Wolfson, T., Trivedi, H., Geva, M., Goldberg, Y., Roth, D., Khot, T., Sabharwal, A., & Tsarfaty, R. (2026). M o N a C o : More Natural and Complex Questions for Reasoning Across Dozens of Documents. Transactions of the Association for Computational Linguistics, 14, 23-46. https://doi.org/10.1162/tacl.a.64
Vancouver
Wolfson T, Trivedi H, Geva M, Goldberg Y, Roth D, Khot T, et al. M o N a C o : More Natural and Complex Questions for Reasoning Across Dozens of Documents. Transactions of the Association for Computational Linguistics. 2026;14:23-46. doi:10.1162/tacl.a.64
BibTeX
@article{Wolfson2026,
author = {Wolfson, Tomer and Trivedi, Harsh and Geva, Mor and Goldberg, Yoav and Roth, Dan and Khot, Tushar and Sabharwal, Ashish and Tsarfaty, Reut},
title = {M o N a C o : More Natural and Complex Questions for Reasoning Across Dozens of Documents},
journal = {Transactions of the Association for Computational Linguistics},
year = {2026},
volume = {14},
pages = {23--46},
doi = {10.1162/tacl.a.64}
}
EUVIMED