EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

DIGEST: Fast and Communication Efficient Decentralized Learning With Local Updates

Peyman Gholami, Hulya Seferoglu

IEEE Transactions on Machine Learning in Communications and Networking · 2024 · 2 · 1456-1474

DOI: 10.1109/tmlcn.2024.3354236 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Machine Learning in Communications and Networking, 2024. Verfasst von Peyman Gholami und 1 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Army Research Laboratory, National Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Two widely considered decentralized learning algorithms are Gossip and random walk-based learning. Gossip algorithms (both synchronous and asynchronous versions) suffer from high communication cost, while random-walk based learning experiences increased convergence time. In this paper, we design a fast and communication-efficient asynchronous decentralized learning mechanism DIGEST by taking advantage of both Gossip and random-walk ideas, and focusing on stochastic gradient descent (SGD). DIGEST is an asynchronous decentralized algorithm building on local-SGD algorithms, which are originally designed for communication efficient centralized learning. We design both single-stream and multi-stream DIGEST, where the communication overhead may increase when the number of streams increases, and there is a convergence and communication overhead trade-off which can be leveraged. We analyze the convergence of single- and multi-stream DIGEST, and prove that both algorithms approach to the optimal solution asymptotically for both iid and non-iid data distributions. We evaluate the performance of single- and multi-stream DIGEST for logistic regression and a deep neural network ResNet20. The simulation results confirm that multi-stream DIGEST has nice convergence properties; i.e., its convergence time is better than or comparable to the baselines in iid setting, and outperforms the baselines in non-iid setting.

Förderung

  • Army Research Laboratory
  • National Science Foundation

Zitieren

APA

Gholami, P., & Seferoglu, H. (2024). DIGEST: Fast and Communication Efficient Decentralized Learning With Local Updates. IEEE Transactions on Machine Learning in Communications and Networking, 2, 1456-1474. https://doi.org/10.1109/tmlcn.2024.3354236

Vancouver

Gholami P, Seferoglu H. DIGEST: Fast and Communication Efficient Decentralized Learning With Local Updates. IEEE Transactions on Machine Learning in Communications and Networking. 2024;2:1456-1474. doi:10.1109/tmlcn.2024.3354236

BibTeX
@article{Gholami2024,
  author = {Gholami, Peyman and Seferoglu, Hulya},
  title = {DIGEST: Fast and Communication Efficient Decentralized Learning With Local Updates},
  journal = {IEEE Transactions on Machine Learning in Communications and Networking},
  year = {2024},
  volume = {2},
  pages = {1456--1474},
  doi = {10.1109/tmlcn.2024.3354236}
}
Kostenlos starten →