EUVIMEDEUVIMED entdecken
Crossref

Randomized matrix sketching for neural network training and gradient monitoring

Harbir Antil, Deepanshu Verma

Mathematical Models and Methods in Applied Sciences · 2026-07 · 36 (08) · 1623-1648

DOI: 10.1142/s0218202526420017 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Mathematical Models and Methods in Applied Sciences, online erschienen am 11.04.2026. Verfasst von Harbir Antil und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 27 Literaturangaben. Gefördert von: Office of Naval Research, NSF, Air Force Office of Scientific Research.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Neural network training relies on gradient computation through backpropagation, yet memory requirements for storing layer activations present significant scalability challenges. We present the first adaptation of control-theoretic matrix sketching to neural network layer activations, enabling memory-efficient gradient reconstruction in backpropagation. This work builds on recent matrix sketching frameworks for dynamic optimization problems, where similar state trajectory storage challenges motivate sketching techniques. Our approach sketches layer activations using three complementary sketch matrices maintained through exponential moving averages with adaptive rank adjustment, automatically balancing memory efficiency against approximation quality. Empirical evaluation on MNIST, CIFAR-10, and physics-informed neural networks demonstrates a controllable accuracy-memory tradeoff. We demonstrate a gradient monitoring application on MNIST showing how sketched activations enable real-time gradient norm tracking with minimal memory overhead. These results establish that sketched activation storage provides a viable path toward memory-efficient neural network training and analysis.

Förderung

  • Office of Naval Research
  • NSF
  • Air Force Office of Scientific Research

Zitieren

APA

Antil, H., & Verma, D. (2026). Randomized matrix sketching for neural network training and gradient monitoring. Mathematical Models and Methods in Applied Sciences, 36(08), 1623-1648. https://doi.org/10.1142/s0218202526420017

Vancouver

Antil H, Verma D. Randomized matrix sketching for neural network training and gradient monitoring. Mathematical Models and Methods in Applied Sciences. 2026;36(08):1623-1648. doi:10.1142/s0218202526420017

BibTeX
@article{Antil2026,
  author = {Antil, Harbir and Verma, Deepanshu},
  title = {Randomized matrix sketching for neural network training and gradient monitoring},
  journal = {Mathematical Models and Methods in Applied Sciences},
  year = {2026},
  volume = {36},
  number = {08},
  pages = {1623--1648},
  doi = {10.1142/s0218202526420017}
}
Kostenlos starten →