EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Asymptotic theory of in-context learning by linear attention

Yue M Lu iD, Mary Letey, Jacob A Zavatone-Veth iD, Anindita Maiti, Cengiz Pehlevan iD

Proceedings of the National Academy of Sciences · 15.07.2025 · 122 (28)

DOI: 10.1073/pnas.2502599122 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the National Academy of Sciences, online erschienen am 09.07.2025. Verfasst von Yue M Lu und 4 weiteren Autor:innen. Beteiligte Einrichtung: The John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 58 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Science Foundation (NSF).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Transformers have a remarkable ability to learn and execute tasks based on examples provided within the input itself, without explicit prior training. It has been argued that this capability, known as in-context learning (ICL), is a cornerstone of Transformers’ success, yet questions about the necessary sample complexity, pretraining task diversity, and context length for successful ICL remain unresolved. Here, we provide a precise answer to these questions in an exactly solvable model of ICL of a linear regression task by linear attention. We derive sharp asymptotics for the learning curve in a phenomenologically rich scaling regime where the token dimension is taken to infinity; the context length and pretraining task diversity scale proportionally with the token dimension; and the number of pretraining examples scales quadratically. We demonstrate a double-descent learning curve with increasing pretraining examples, and uncover a phase transition in the model’s behavior between low and high task diversity regimes: in the low diversity regime, the model tends toward memorization of training tasks, whereas in the high diversity regime, it achieves genuine ICL and generalization beyond the scope of pretrained tasks. These theoretical insights are empirically validated through experiments with both linear attention and full nonlinear Transformer architectures.

Schlagwörter

Stichwörter der Autor:innen

in-context learningmachine learningtransformers

Autor:innen und Institutionen

  • The John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138.
  • Center for Brain Science, Harvard University, Cambridge, MA 02138.
  • Society of Fellows, Harvard University, Cambridge, MA 02138.
  • Department of Physics, Harvard University, Cambridge, MA 02138.
  • Perimeter Institute for Theoretical Physics, Waterloo, ON N2L 2Y5, Canada.
  • The Kempner Institute for the Study of Natural and Artificial Intelligence, 150 Western Avenue, Alston, MA 02134.

ORCID-Profile: Yue M Lu, Jacob A Zavatone-Veth, Cengiz Pehlevan

Förderung

  • National Science Foundation (NSF)

Zitieren

APA

Lu, Y. M., Letey, M., Zavatone-Veth, J. A., Maiti, A., & Pehlevan, C. (2025). Asymptotic theory of in-context learning by linear attention. Proceedings of the National Academy of Sciences, 122(28). https://doi.org/10.1073/pnas.2502599122

Vancouver

Lu YM, Letey M, Zavatone-Veth JA, Maiti A, Pehlevan C. Asymptotic theory of in-context learning by linear attention. Proceedings of the National Academy of Sciences. 2025;122(28). doi:10.1073/pnas.2502599122

BibTeX
@article{Lu2025,
  author = {Lu, Yue M and Letey, Mary and Zavatone-Veth, Jacob A and Maiti, Anindita and Pehlevan, Cengiz},
  title = {Asymptotic theory of in-context learning by linear attention},
  journal = {Proceedings of the National Academy of Sciences},
  year = {2025},
  volume = {122},
  number = {28},
  doi = {10.1073/pnas.2502599122}
}
Kostenlos starten →