Mapping of attention mechanisms to a generalized Potts model
Physical Review Research · 16.04.2024 · 6 (2)
DOI: 10.1103/physrevresearch.6.023057 ↗Kurz gefasst
Fachartikel in der Zeitschrift Physical Review Research, online erschienen am 16.04.2024. Verfasst von Riccardo Rende und 3 weiteren Autor:innen. Die Arbeit wurde bisher 37-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Transformers are neural networks that revolutionized natural language processing and machine learning. They process sequences of inputs, like words, using a mechanism called self-attention, which is trained via masked language modeling (MLM). In MLM, a word is randomly masked in an input sequence, and the network is trained to predict the missing word. Despite the practical success of transformers, it remains unclear what type of data distribution self-attention can learn efficiently. Here, we show analytically that if one decouples the treatment of word positions and embeddings, a single layer of self-attention learns the conditionals of a generalized Potts model with interactions between sites and Potts colors. Moreover, we show that training this neural network is exactly equivalent to solving the inverse Potts problem by the so-called pseudolikelihood method, well known in statistical physics. Using this mapping, we compute the generalization error of self-attention in a model scenario analytically using the replica method. Published by the American Physical Society 2024
Zitieren
APA
Rende, R., Gerace, F., Laio, A., & Goldt, S. (2024). Mapping of attention mechanisms to a generalized Potts model. Physical Review Research, 6(2). https://doi.org/10.1103/physrevresearch.6.023057
Vancouver
Rende R, Gerace F, Laio A, Goldt S. Mapping of attention mechanisms to a generalized Potts model. Physical Review Research. 2024;6(2). doi:10.1103/physrevresearch.6.023057
BibTeX
@article{Rende2024,
author = {Rende, Riccardo and Gerace, Federica and Laio, Alessandro and Goldt, Sebastian},
title = {Mapping of attention mechanisms to a generalized Potts model},
journal = {Physical Review Research},
year = {2024},
volume = {6},
number = {2},
doi = {10.1103/physrevresearch.6.023057}
}
EUVIMED