EUVIMEDEUVIMED entdecken
Crossref

Greedy Low-Rank Gradient Compression for Distributed Learning With Convergence Guarantees

Chuyan Chen, Yutong He, Pengrui Li, Weichen Jia, Kun Yuan

IEEE Transactions on Signal Processing · 2026 · 74 · 937-951

DOI: 10.1109/tsp.2026.3667669 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Signal Processing, 2026. Verfasst von Chuyan Chen und 4 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 61 Literaturangaben. Gefördert von: NSF China, National Key Research and Development Program of China, National Engineering Labratory for Big Data Analytics and Applications, and AI for Science Institute, Beijing, China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Distributed optimization is pivotal for large-scale signal processing and machine learning, yet communication overhead remains a major bottleneck. Low-rank gradient compression, in which the transmitted gradients are approximated by low-rank matrices to reduce communication, offers a promising remedy. Existing methods typically adopt either randomized or greedy compression strategies: randomized approaches project gradients onto randomly chosen subspaces, introducing high variance and degrading empirical performance; greedy methods select the most informative subspaces, achieving strong empirical results but lacking convergence guarantees. To address this gap, we propose GreedyLore—the first Greedy Lo w- R ank gradi e nt compression algorithm for distributed learning with rigorous convergence guarantees. GreedyLore incorporates error feedback to correct the bias introduced by greedy compression and introduces a semi-lazy subspace update that ensures the compression operator remains contractive throughout all iterations. With these techniques, we prove that GreedyLore achieves a convergence rate of $\mathcal{O}(\sigma/\sqrt{NT}+1/T)$ under standard optimizers such as MSGD and Adam—marking the first linear speedup convergence rate for low-rank gradient compression. Extensive experiments are conducted to validate our theoretical findings.

Förderung

  • NSF China
  • National Key Research and Development Program of China
  • National Engineering Labratory for Big Data Analytics and Applications, and AI for Science Institute, Beijing, China

Zitieren

APA

Chen, C., He, Y., Li, P., Jia, W., & Yuan, K. (2026). Greedy Low-Rank Gradient Compression for Distributed Learning With Convergence Guarantees. IEEE Transactions on Signal Processing, 74, 937-951. https://doi.org/10.1109/tsp.2026.3667669

Vancouver

Chen C, He Y, Li P, Jia W, Yuan K. Greedy Low-Rank Gradient Compression for Distributed Learning With Convergence Guarantees. IEEE Transactions on Signal Processing. 2026;74:937-951. doi:10.1109/tsp.2026.3667669

BibTeX
@article{Chen2026,
  author = {Chen, Chuyan and He, Yutong and Li, Pengrui and Jia, Weichen and Yuan, Kun},
  title = {Greedy Low-Rank Gradient Compression for Distributed Learning With Convergence Guarantees},
  journal = {IEEE Transactions on Signal Processing},
  year = {2026},
  volume = {74},
  pages = {937--951},
  doi = {10.1109/tsp.2026.3667669}
}
Kostenlos starten →