EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Quantization avoids saddle points in distributed optimization

Yanan Bo iD, Yongqiang Wang

Proceedings of the National Academy of Sciences · 23.04.2024 · 121 (17)

DOI: 10.1073/pnas.2319625121 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the National Academy of Sciences, online erschienen am 19.04.2024. Verfasst von Yanan Bo und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Electrical and Computer Engineering, Clemson University, Clemson, SC 29634. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: NSF | CISE | Division of Computing and Communication Foundations (CCF).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Distributed nonconvex optimization underpins key functionalities of numerous distributed systems, ranging from power systems, smart buildings, cooperative robots, vehicle networks to sensor networks. Recently, it has also merged as a promising solution to handle the enormous growth in data and model sizes in deep learning. A fundamental problem in distributed nonconvex optimization is avoiding convergence to saddle points, which significantly degrade optimization accuracy. We find that the process of quantization, which is necessary for all digital communications, can be exploited to enable saddle-point avoidance. More specifically, we propose a stochastic quantization scheme and prove that it can effectively escape saddle points and ensure convergence to a second-order stationary point in distributed nonconvex optimization. With an easily adjustable quantization granularity, the approach allows a user to control the number of bits sent per iteration and, hence, to aggressively reduce the communication overhead. Numerical experimental results using distributed optimization and learning problems on benchmark datasets confirm the effectiveness of the approach.

Schlagwörter

Stichwörter der Autor:innen

distributed nonconvex optimizationquantizationsaddle-point avoidance

Autor:innen und Institutionen

  • Department of Electrical and Computer Engineering, Clemson University, Clemson, SC 29634.

ORCID-Profile: Yanan Bo

Förderung

  • NSF | CISE | Division of Computing and Communication Foundations (CCF)

Zitieren

APA

Bo, Y., & Wang, Y. (2024). Quantization avoids saddle points in distributed optimization. Proceedings of the National Academy of Sciences, 121(17). https://doi.org/10.1073/pnas.2319625121

Vancouver

Bo Y, Wang Y. Quantization avoids saddle points in distributed optimization. Proceedings of the National Academy of Sciences. 2024;121(17). doi:10.1073/pnas.2319625121

BibTeX
@article{Bo2024,
  author = {Bo, Yanan and Wang, Yongqiang},
  title = {Quantization avoids saddle points in distributed optimization},
  journal = {Proceedings of the National Academy of Sciences},
  year = {2024},
  volume = {121},
  number = {17},
  doi = {10.1073/pnas.2319625121}
}
Kostenlos starten →