Stable tensor neural networks for efficient deep learning
Frontiers in Big Data · 30.05.2024 · 7
DOI: 10.3389/fdata.2024.1363978 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Big Data, online erschienen am 30.05.2024. Verfasst von Elizabeth Newman und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Mathematics, Emory University, Atlanta, GA, United States. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 55 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Learning from complex, multidimensional data has become central to computational mathematics, and among the most successful high-dimensional function approximators are deep neural networks (DNNs). Training DNNs is posed as an optimization problem to learn network weights or parameters that well-approximate a mapping from input to target data. Multiway data or tensors arise naturally in myriad ways in deep learning, in particular as input data and as high-dimensional weights and features extracted by the network, with the latter often being a bottleneck in terms of speed and memory. In this work, we leverage tensor representations and processing to efficiently parameterize DNNs when learning from high-dimensional data. We propose tensor neural networks (t-NNs), a natural extension of traditional fully-connected networks, that can be trained efficiently in a reduced, yet more powerful parameter space. Our t-NNs are built upon matrix-mimetic tensor-tensor products, which retain algebraic properties of matrix multiplication while capturing high-dimensional correlations. Mimeticity enables t-NNs to inherit desirable properties of modern DNN architectures. We exemplify this by extending recent work on stable neural networks, which interpret DNNs as discretizations of differential equations, to our multidimensional framework. We provide empirical evidence of the parametric advantages of t-NNs on dimensionality reduction using autoencoders and classification using fully-connected and stable variants on benchmark imaging datasets MNIST and CIFAR-10.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Mathematics, Emory University, Atlanta, GA, United States.
- Mathematics and Theoretical Computer Science, IBM TJ Watson Research Center, Yorktown, NY, United States.
- Department of Applied Mathematics, Tel Aviv University, Tel Aviv-Yafo, Israel.
- Department of Mathematics, Tufts University, Medford, MA, United States.
Zitieren
APA
Newman, E., Horesh, L., Avron, H., & Kilmer, M. E. (2024). Stable tensor neural networks for efficient deep learning. Frontiers in Big Data, 7. https://doi.org/10.3389/fdata.2024.1363978
Vancouver
Newman E, Horesh L, Avron H, Kilmer ME. Stable tensor neural networks for efficient deep learning. Frontiers in Big Data. 2024;7. doi:10.3389/fdata.2024.1363978
BibTeX
@article{Newman2024,
author = {Newman, Elizabeth and Horesh, Lior and Avron, Haim and Kilmer, Misha E},
title = {Stable tensor neural networks for efficient deep learning},
journal = {Frontiers in Big Data},
year = {2024},
volume = {7},
doi = {10.3389/fdata.2024.1363978}
}
EUVIMED