EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

On Memorization and Generalization in Compact Transformers

Aki Härmä, Ali Al-Saeedi, Anton Changalidis, Dumitru Verşebeniuc, Marcin Pietrasik, Anna Wilbik

Electronics · 27.04.2026 · 15 (9) · 1847

DOI: 10.3390/electronics15091847 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 27.04.2026. Verfasst von Aki Härmä und 5 weiteren Autor:innen. Die Arbeit nennt 97 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) seem to demonstrate human-like understanding and generalization of language content. These arise from the capabilities of the models to memorize and generalize the training content. In this paper, we review the recent literature and theories on the mechanisms in self-attention neural networks. We also report three computational experiments that give insights into the underlying mechanisms and capabilities of the models. We also report three computational experiments showing that memorization capacity in compact transformers can be empirically linked to architectural parameters, that structured domain knowledge can be retained in small decoder-only models, and that in-context abstraction requires sufficient architectural depth. These findings suggest that the current models are superfluous for many specific applications, especially in on-edge use cases. A better understanding of application requirements and architecture details can be expected to help in building new LLM architectures that can be efficiently implemented on dedicated on-edge circuits.

Zitieren

APA

Härmä, A., Al-Saeedi, A., Changalidis, A., Verşebeniuc, D., Pietrasik, M., & Wilbik, A. (2026). On Memorization and Generalization in Compact Transformers. Electronics, 15(9), 1847. https://doi.org/10.3390/electronics15091847

Vancouver

Härmä A, Al-Saeedi A, Changalidis A, Verşebeniuc D, Pietrasik M, Wilbik A. On Memorization and Generalization in Compact Transformers. Electronics. 2026;15(9):1847. doi:10.3390/electronics15091847

BibTeX
@article{Hrm2026,
  author = {Härmä, Aki and Al-Saeedi, Ali and Changalidis, Anton and Verşebeniuc, Dumitru and Pietrasik, Marcin and Wilbik, Anna},
  title = {On Memorization and Generalization in Compact Transformers},
  journal = {Electronics},
  year = {2026},
  volume = {15},
  number = {9},
  pages = {1847},
  doi = {10.3390/electronics15091847}
}
Kostenlos starten →