EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

SPSA View on the Straight-Through Estimator in Neural Network Quantization

Sergey Salishev, Anton Makarov, Elizaveta Tarasova, Kirill Shishin, Oleg Granichin

IEEE Access · 2026 · 14 · 60047-60057

DOI: 10.1109/access.2026.3683267 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Access, 2026. Verfasst von Sergey Salishev und 4 weiteren Autor:innen. Die Arbeit nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Economic Development of the Russian Federation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Low-bit quantization-aware training of neural networks typically relies on the straight-through estimator (STE) to learn both quantized weights and their associated scales or effective bit-widths. Building on earlier analysis of gradual differentiable noise-scale quantization, we first recall that the classical STE noise-scale gradient can be interpreted as an asymptotically optimal estimator of the rounding-noise gradient obtained by averaging over mini-batches. This viewpoint justifies replacing the exact rounding residual by any zero-mean proxy noise with matched variance, such as symmetric Bernoulli (Rademacher), Uniform or Gaussian perturbations. In this paper we make the connection between such STE-based scale updates and simultaneous perturbation stochastic approximation (SPSA) precise. We show that injecting noise into the learnable quantization scale implements a one-measurement SPSA scheme in the low-dimensional noise-scale space, and we verify the standard stochastic-approximation assumptions underlying the local convergence analysis for a realistic quantization-aware loss with a bit-width penalty. Experiments with one-bit weight and activation quantization on ResNet-20 for CIFAR-10/100 classification, and four-bit quantization on RFDN for standard super-resolution (SR) benchmarks, demonstrate that the stochastic scale update consistently accelerates convergence of the effective bit-width while preserving final accuracy compared to empirical rounding. In a PyTorch implementation this modification requires only one additional line on top of a standard LSQ-type quantization-aware training loop, adding negligible computational and memory overhead.

Förderung

  • Ministry of Economic Development of the Russian Federation

Zitieren

APA

Salishev, S., Makarov, A., Tarasova, E., Shishin, K., & Granichin, O. (2026). SPSA View on the Straight-Through Estimator in Neural Network Quantization. IEEE Access, 14, 60047-60057. https://doi.org/10.1109/access.2026.3683267

Vancouver

Salishev S, Makarov A, Tarasova E, Shishin K, Granichin O. SPSA View on the Straight-Through Estimator in Neural Network Quantization. IEEE Access. 2026;14:60047-60057. doi:10.1109/access.2026.3683267

BibTeX
@article{Salishev2026,
  author = {Salishev, Sergey and Makarov, Anton and Tarasova, Elizaveta and Shishin, Kirill and Granichin, Oleg},
  title = {SPSA View on the Straight-Through Estimator in Neural Network Quantization},
  journal = {IEEE Access},
  year = {2026},
  volume = {14},
  pages = {60047--60057},
  doi = {10.1109/access.2026.3683267}
}
Kostenlos starten →