Greedy Low-Rank Gradient Compression for Distributed Learning With Convergence Guarantees
IEEE Transactions on Signal Processing · 2026 · 74 · 937-951
DOI: 10.1109/tsp.2026.3667669 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Signal Processing, 2026. Verfasst von Chuyan Chen und 4 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 61 Literaturangaben. Gefördert von: NSF China, National Key Research and Development Program of China, National Engineering Labratory for Big Data Analytics and Applications, and AI for Science Institute, Beijing, China.
Abstract (Originalsprache)
Distributed optimization is pivotal for large-scale signal processing and machine learning, yet communication overhead remains a major bottleneck. Low-rank gradient compression, in which the transmitted gradients are approximated by low-rank matrices to reduce communication, offers a promising remedy. Existing methods typically adopt either randomized or greedy compression strategies: randomized approaches project gradients onto randomly chosen subspaces, introducing high variance and degrading empirical performance; greedy methods select the most informative subspaces, achieving strong empirical results but lacking convergence guarantees. To address this gap, we propose GreedyLore—the first Greedy Lo w- R ank gradi e nt compression algorithm for distributed learning with rigorous convergence guarantees. GreedyLore incorporates error feedback to correct the bias introduced by greedy compression and introduces a semi-lazy subspace update that ensures the compression operator remains contractive throughout all iterations. With these techniques, we prove that GreedyLore achieves a convergence rate of $\mathcal{O}(\sigma/\sqrt{NT}+1/T)$ under standard optimizers such as MSGD and Adam—marking the first linear speedup convergence rate for low-rank gradient compression. Extensive experiments are conducted to validate our theoretical findings.
Förderung
- NSF China
- National Key Research and Development Program of China
- National Engineering Labratory for Big Data Analytics and Applications, and AI for Science Institute, Beijing, China
Zitieren
APA
Chen, C., He, Y., Li, P., Jia, W., & Yuan, K. (2026). Greedy Low-Rank Gradient Compression for Distributed Learning With Convergence Guarantees. IEEE Transactions on Signal Processing, 74, 937-951. https://doi.org/10.1109/tsp.2026.3667669
Vancouver
Chen C, He Y, Li P, Jia W, Yuan K. Greedy Low-Rank Gradient Compression for Distributed Learning With Convergence Guarantees. IEEE Transactions on Signal Processing. 2026;74:937-951. doi:10.1109/tsp.2026.3667669
BibTeX
@article{Chen2026,
author = {Chen, Chuyan and He, Yutong and Li, Pengrui and Jia, Weichen and Yuan, Kun},
title = {Greedy Low-Rank Gradient Compression for Distributed Learning With Convergence Guarantees},
journal = {IEEE Transactions on Signal Processing},
year = {2026},
volume = {74},
pages = {937--951},
doi = {10.1109/tsp.2026.3667669}
}
EUVIMED