Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity
Transactions of the Association for Computational Linguistics · 01.06.2026 · 14 · 936-959
DOI: 10.1162/tacl.a.691 ↗Kurz gefasst
Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 01.06.2026. Verfasst von Pamela D. Rivière und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 51 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Despite an in-principle understanding of self-attention matrix operations in Transformer language models (LMs), it remains unclear precisely how these operations map onto interpretable computations or functions—and how or when individual attention heads develop specialized attention patterns. Here, we present a pipeline to systematically probe attention mechanisms, and we illustrate its value by leveraging lexical ambiguity—where a single word has multiple meanings—to isolate attention mechanisms that contribute to word sense disambiguation. We take a “developmental” approach: first, using publicly available Pythia LM checkpoints, we identify inflection points in disambiguation performance for each LM in the suite; in 14M and 410M, we identify heads whose attention to disambiguating words covaries with overall disambiguation performance across development. We then stress-test the robustness of these heads to stimulus perturbations: in 14M, we find limited robustness, but in 410M, we identify multiple heads with surprisingly generalizable behavior. Then, in a causal analysis, we find that ablating the target heads demonstrably impairs disambiguation performance, particularly in 14M. We additionally reproduce developmental analyses of 14M across all of its random seeds. Together, these results suggest: that disambiguation benefits from a constellation of mechanisms, some of which (especially in 14M) are highly sensitive to the position and part-of-speech of the disambiguating cue; and that larger models (410M) may contain heads with more robust disambiguation behavior. They also join a growing body of work that highlights the value of adopting a developmental perspective when probing LM mechanisms.
Zitieren
APA
Rivière, P. D., & Trott, S. (2026). Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity. Transactions of the Association for Computational Linguistics, 14, 936-959. https://doi.org/10.1162/tacl.a.691
Vancouver
Rivière PD, Trott S. Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity. Transactions of the Association for Computational Linguistics. 2026;14:936-959. doi:10.1162/tacl.a.691
BibTeX
@article{Rivire2026,
author = {Rivière, Pamela D. and Trott, Sean},
title = {Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity},
journal = {Transactions of the Association for Computational Linguistics},
year = {2026},
volume = {14},
pages = {936--959},
doi = {10.1162/tacl.a.691}
}
EUVIMED