How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
Physical Review X · 01.07.2024 · 14 (3)
DOI: 10.1103/physrevx.14.031001 ↗Kurz gefasst
Fachartikel in der Zeitschrift Physical Review X, online erschienen am 01.07.2024. Verfasst von Francesco Cagnetta und 4 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 52 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Simons Foundation.
Abstract (Originalsprache)
Deep learning algorithms demonstrate a surprising ability to learn high-dimensional tasks from limited examples. This is commonly attributed to the depth of neural networks, enabling them to build a hierarchy of abstract, low-dimensional data representations. However, how many training examples are required to learn such representations remains unknown. To quantitatively study this question, we introduce the random hierarchy model: a family of synthetic tasks inspired by the hierarchical structure of language and images. The model is a classification task where each class corresponds to a group of high-level features, chosen among several equivalent groups associated with the same class. In turn, each feature corresponds to a group of subfeatures chosen among several equivalent groups and so on, following a hierarchy of composition rules. We find that deep networks learn the task by developing internal representations invariant to exchanging equivalent groups. Moreover, the number of data required corresponds to the point where correlations between low-level features and classes become detectable. Overall, our results indicate how deep networks overcome the curse of dimensionality by building invariant representations and provide an estimate of the number of data required to learn a hierarchical task. Published by the American Physical Society 2024
Förderung
- Simons Foundation
Zitieren
APA
Cagnetta, F., Petrini, L., Tomasini, U. M., Favero, A., & Wyart, M. (2024). How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model. Physical Review X, 14(3). https://doi.org/10.1103/physrevx.14.031001
Vancouver
Cagnetta F, Petrini L, Tomasini UM, Favero A, Wyart M. How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model. Physical Review X. 2024;14(3). doi:10.1103/physrevx.14.031001
BibTeX
@article{Cagnetta2024,
author = {Cagnetta, Francesco and Petrini, Leonardo and Tomasini, Umberto M. and Favero, Alessandro and Wyart, Matthieu},
title = {How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model},
journal = {Physical Review X},
year = {2024},
volume = {14},
number = {3},
doi = {10.1103/physrevx.14.031001}
}
EUVIMED