Mechanism for feature learning in neural networks and backpropagation-free machine learning models
Science · 29.03.2024 · 383 (6690) · 1461-1467
DOI: 10.1126/science.adi5639 ↗Kurz gefasst
Fachartikel in der Zeitschrift Science, erschienen am 29.03.2024. Verfasst von Adityanarayanan Radhakrishnan und 3 weiteren Autor:innen. Beteiligte Einrichtung: Harvard School of Engineering and Applied Sciences, Cambridge, MA 02138, USA. Die Arbeit wurde bisher 81-mal zitiert (Crossref) und nennt 72 Literaturangaben.
Abstract (Originalsprache)
Understanding how neural networks learn features, or relevant patterns in data, for prediction is necessary for their reliable use in technological and scientific applications. In this work, we presented a unifying mathematical mechanism, known as average gradient outer product (AGOP), that characterized feature learning in neural networks. We provided empirical evidence that AGOP captured features learned by various neural network architectures, including transformer-based language models, convolutional networks, multilayer perceptrons, and recurrent neural networks. Moreover, we demonstrated that AGOP, which is backpropagation-free, enabled feature learning in machine learning models, such as kernel machines, that a priori could not identify task-specific features. Overall, we established a fundamental mechanism that captured feature learning in neural networks and enabled feature learning in general machine learning models.
Autor:innen und Institutionen
- Harvard School of Engineering and Applied Sciences, Cambridge, MA 02138, USA.
- Broad Institute of MIT and Harvard, Cambridge, MA 02142, USA.
- Computer Science and Engineering, UC San Diego, La Jolla, CA 92093, USA.
- Center for Machine Intelligence and Data Science, IIT Bombay, Mumbai 400076, India.
- Halıcıoğlu Data Science Institute, UC San Diego, La Jolla, CA 92093, USA.
ORCID-Profile: Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, Mikhail Belkin
Zitieren
APA
Radhakrishnan, A., Beaglehole, D., Pandit, P., & Belkin, M. (2024). Mechanism for feature learning in neural networks and backpropagation-free machine learning models. Science, 383(6690), 1461-1467. https://doi.org/10.1126/science.adi5639
Vancouver
Radhakrishnan A, Beaglehole D, Pandit P, Belkin M. Mechanism for feature learning in neural networks and backpropagation-free machine learning models. Science. 2024;383(6690):1461-1467. doi:10.1126/science.adi5639
BibTeX
@article{Radhakrishnan2024,
author = {Radhakrishnan, Adityanarayanan and Beaglehole, Daniel and Pandit, Parthe and Belkin, Mikhail},
title = {Mechanism for feature learning in neural networks and backpropagation-free machine learning models},
journal = {Science},
year = {2024},
volume = {383},
number = {6690},
pages = {1461--1467},
doi = {10.1126/science.adi5639}
}
EUVIMED