SPSA View on the Straight-Through Estimator in Neural Network Quantization
IEEE Access · 2026 · 14 · 60047-60057
DOI: 10.1109/access.2026.3683267 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Access, 2026. Verfasst von Sergey Salishev und 4 weiteren Autor:innen. Die Arbeit nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Economic Development of the Russian Federation.
Abstract (Originalsprache)
Low-bit quantization-aware training of neural networks typically relies on the straight-through estimator (STE) to learn both quantized weights and their associated scales or effective bit-widths. Building on earlier analysis of gradual differentiable noise-scale quantization, we first recall that the classical STE noise-scale gradient can be interpreted as an asymptotically optimal estimator of the rounding-noise gradient obtained by averaging over mini-batches. This viewpoint justifies replacing the exact rounding residual by any zero-mean proxy noise with matched variance, such as symmetric Bernoulli (Rademacher), Uniform or Gaussian perturbations. In this paper we make the connection between such STE-based scale updates and simultaneous perturbation stochastic approximation (SPSA) precise. We show that injecting noise into the learnable quantization scale implements a one-measurement SPSA scheme in the low-dimensional noise-scale space, and we verify the standard stochastic-approximation assumptions underlying the local convergence analysis for a realistic quantization-aware loss with a bit-width penalty. Experiments with one-bit weight and activation quantization on ResNet-20 for CIFAR-10/100 classification, and four-bit quantization on RFDN for standard super-resolution (SR) benchmarks, demonstrate that the stochastic scale update consistently accelerates convergence of the effective bit-width while preserving final accuracy compared to empirical rounding. In a PyTorch implementation this modification requires only one additional line on top of a standard LSQ-type quantization-aware training loop, adding negligible computational and memory overhead.
Förderung
- Ministry of Economic Development of the Russian Federation
Zitieren
APA
Salishev, S., Makarov, A., Tarasova, E., Shishin, K., & Granichin, O. (2026). SPSA View on the Straight-Through Estimator in Neural Network Quantization. IEEE Access, 14, 60047-60057. https://doi.org/10.1109/access.2026.3683267
Vancouver
Salishev S, Makarov A, Tarasova E, Shishin K, Granichin O. SPSA View on the Straight-Through Estimator in Neural Network Quantization. IEEE Access. 2026;14:60047-60057. doi:10.1109/access.2026.3683267
BibTeX
@article{Salishev2026,
author = {Salishev, Sergey and Makarov, Anton and Tarasova, Elizaveta and Shishin, Kirill and Granichin, Oleg},
title = {SPSA View on the Straight-Through Estimator in Neural Network Quantization},
journal = {IEEE Access},
year = {2026},
volume = {14},
pages = {60047--60057},
doi = {10.1109/access.2026.3683267}
}
EUVIMED