Data Foundations of Long-Context Language Models: A Survey
Transactions of the Association for Computational Linguistics · 22.07.2026 · 14 · 1803-1825
DOI: 10.1162/tacl.a.775 ↗Kurz gefasst
Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 22.07.2026. Verfasst von Zechen Sun und 7 weiteren Autor:innen. Die Arbeit nennt 159 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
As the context window of Large Language Models (LLMs) continues to expand, the data required to effectively train and evaluate these capabilities remains underexplored. With existing research primarily focuses on architectural optimization, there is a need for a systematic, data-centric review. This survey bridges this gap by investigating the data foundations of Long-Context Language Models (LCMs). We begin by examining current data strategies alongside their strengths and limitations, mapping the required data to desired model capabilities. Building on this, we explore how targeted training data designs drive core, often interconnected skills such as retrieval, reasoning, and aggregation. Furthermore, we analyze the evaluation landscape, illustrating how selecting appropriate benchmarks is crucial for probing capability boundaries and guiding effective model selection. Finally, we synthesize actionable guidelines for data construction and outline critical future directions to propel the advancement of long-context language models, including quantifying data quality, establishing scaling laws for length distributions, and developing dynamic evaluation frameworks.
Zitieren
APA
Sun, Z., Sun, Y., Su, Z., Tang, Z., Li, J., Zhou, A., Chen, W., & Zhang, M. (2026). Data Foundations of Long-Context Language Models: A Survey. Transactions of the Association for Computational Linguistics, 14, 1803-1825. https://doi.org/10.1162/tacl.a.775
Vancouver
Sun Z, Sun Y, Su Z, Tang Z, Li J, Zhou A, et al. Data Foundations of Long-Context Language Models: A Survey. Transactions of the Association for Computational Linguistics. 2026;14:1803-1825. doi:10.1162/tacl.a.775
BibTeX
@article{Sun2026,
author = {Sun, Zechen and Sun, Yuyang and Su, Zhaochen and Tang, Zecheng and Li, Juntao and Zhou, Ao and Chen, Wenliang and Zhang, Min},
title = {Data Foundations of Long-Context Language Models: A Survey},
journal = {Transactions of the Association for Computational Linguistics},
year = {2026},
volume = {14},
pages = {1803--1825},
doi = {10.1162/tacl.a.775}
}
EUVIMED