EUVIMEDEUVIMED entdecken
Crossref

Smoothed Gradient Clipping and Error Feedback for Decentralized Optimization under Symmetric Heavy-Tailed Noise

Shuhua Yu, Dušan Jakovetic, Soummya Kar

SIAM Journal on Optimization · 30.06.2026 · 36 (2) · 703-728

DOI: 10.1137/24m170747x ↗

Kurz gefasst

Fachartikel in der Zeitschrift SIAM Journal on Optimization, online erschienen am 23.04.2026. Verfasst von Shuhua Yu und 2 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 49 Literaturangaben. Gefördert von: NSF, Ministry of Science, Technological Development and Innovation, Republic of Serbia, Provincial Secretariat for Higher Education and Scientific Research.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Motivated by understanding and analysis of large-scale machine learning under heavy-tailed gradient noise, we study decentralized optimization with gradient clipping, i.e., in which certain clipping operators are applied to the gradients or gradient estimates computed from local nodes prior to further processing. While vanilla gradient clipping has proven effective in mitigating the impact of heavy-tailed gradient noise in nondistributed setups, it incurs bias that causes convergence issues in heterogeneous distributed settings. To address the inherent bias introduced by gradient clipping, we develop a smoothed clipping operator, and propose a decentralized gradient method equipped with an error feedback mechanism, i.e., the clipping operator is applied on the difference between some local gradient estimator and local stochastic gradient. We consider strongly convex and smooth local functions under symmetric heavy-tailed gradient noise that may not have finite moments of order greater than one. We show that the proposed decentralized gradient clipping method achieves a mean-square error (MSE) convergence rate of [Formula: see text], [Formula: see text], where the exponent [Formula: see text] is independent of the existence of higher order gradient noise moments [Formula: see text] and lower bounded by some constant dependent on condition number. To the best of our knowledge, this is the first MSE convergence result for decentralized gradient clipping under heavy-tailed noise without assuming bounded gradient. Numerical experiments validate our theoretical findings.

Förderung

  • NSF
  • Ministry of Science, Technological Development and Innovation, Republic of Serbia
  • Provincial Secretariat for Higher Education and Scientific Research

Zitieren

APA

Yu, S., Jakovetic, D., & Kar, S. (2026). Smoothed Gradient Clipping and Error Feedback for Decentralized Optimization under Symmetric Heavy-Tailed Noise. SIAM Journal on Optimization, 36(2), 703-728. https://doi.org/10.1137/24m170747x

Vancouver

Yu S, Jakovetic D, Kar S. Smoothed Gradient Clipping and Error Feedback for Decentralized Optimization under Symmetric Heavy-Tailed Noise. SIAM Journal on Optimization. 2026;36(2):703-728. doi:10.1137/24m170747x

BibTeX
@article{Yu2026,
  author = {Yu, Shuhua and Jakovetic, Dušan and Kar, Soummya},
  title = {Smoothed Gradient Clipping and Error Feedback for Decentralized Optimization under Symmetric Heavy-Tailed Noise},
  journal = {SIAM Journal on Optimization},
  year = {2026},
  volume = {36},
  number = {2},
  pages = {703--728},
  doi = {10.1137/24m170747x}
}
Kostenlos starten →