EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Data Foundations of Long-Context Language Models: A Survey

Zechen Sun, Yuyang Sun, Zhaochen Su, Zecheng Tang, Juntao Li, Ao Zhou, Wenliang Chen, Min Zhang

Transactions of the Association for Computational Linguistics · 22.07.2026 · 14 · 1803-1825

DOI: 10.1162/tacl.a.775 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 22.07.2026. Verfasst von Zechen Sun und 7 weiteren Autor:innen. Die Arbeit nennt 159 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

As the context window of Large Language Models (LLMs) continues to expand, the data required to effectively train and evaluate these capabilities remains underexplored. With existing research primarily focuses on architectural optimization, there is a need for a systematic, data-centric review. This survey bridges this gap by investigating the data foundations of Long-Context Language Models (LCMs). We begin by examining current data strategies alongside their strengths and limitations, mapping the required data to desired model capabilities. Building on this, we explore how targeted training data designs drive core, often interconnected skills such as retrieval, reasoning, and aggregation. Furthermore, we analyze the evaluation landscape, illustrating how selecting appropriate benchmarks is crucial for probing capability boundaries and guiding effective model selection. Finally, we synthesize actionable guidelines for data construction and outline critical future directions to propel the advancement of long-context language models, including quantifying data quality, establishing scaling laws for length distributions, and developing dynamic evaluation frameworks.

Zitieren

APA

Sun, Z., Sun, Y., Su, Z., Tang, Z., Li, J., Zhou, A., Chen, W., & Zhang, M. (2026). Data Foundations of Long-Context Language Models: A Survey. Transactions of the Association for Computational Linguistics, 14, 1803-1825. https://doi.org/10.1162/tacl.a.775

Vancouver

Sun Z, Sun Y, Su Z, Tang Z, Li J, Zhou A, et al. Data Foundations of Long-Context Language Models: A Survey. Transactions of the Association for Computational Linguistics. 2026;14:1803-1825. doi:10.1162/tacl.a.775

BibTeX
@article{Sun2026,
  author = {Sun, Zechen and Sun, Yuyang and Su, Zhaochen and Tang, Zecheng and Li, Juntao and Zhou, Ao and Chen, Wenliang and Zhang, Min},
  title = {Data Foundations of Long-Context Language Models: A Survey},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  volume = {14},
  pages = {1803--1825},
  doi = {10.1162/tacl.a.775}
}
Kostenlos starten →