EUVIMEDEUVIMED entdecken
Crossref

No Worker Left (Too Far) Behind: Dynamic Hybrid Synchronization for In‐Network ML Aggregation

Diego Cardoso Nunes, Bruno Loureiro Coelho, Ricardo Parizotto, Alberto Egon Schaeffer‐Filho

International Journal of Network Management · 2025-01 · 35 (1)

DOI: 10.1002/nem.2290 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Network Management, online erschienen am 24.07.2024. Verfasst von Diego Cardoso Nunes und 3 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 53 Literaturangaben. Gefördert von: Conselho Nacional de Desenvolvimento Científico e Tecnológico, Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, Fundação de Amparo à Pesquisa do Estado de São Paulo.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Achieving high‐performance aggregation is essential to scaling data‐parallel distributed machine learning (ML) training. Recent research in in‐network computing has shown that offloading the aggregation to the network data plane can accelerate the aggregation process compared to traditional server‐only approaches, reducing the propagation delay and consequently speeding up distributed training. However, the existing literature on in‐network aggregation does not provide ways to deal with slower workers (called stragglers). The presence of stragglers can negatively impact distributed training, increasing the time it takes to complete. In this paper, we present Serene, an in‐network aggregation system capable of circumventing the effects of stragglers. Serene coordinates the ML workers to cooperate with a programmable switch using a hybrid synchronization approach where approaches can be changed dynamically. The synchronization can change dynamically through a control plane API that translates high‐level code into switch rules. Serene switch employs an efficient data structure for managing synchronization and a hot‐swapping mechanism to consistently change from one synchronization strategy to another. We implemented and evaluated a prototype using BMv2 and a Proof‐of‐Concept in a Tofino ASIC. We ran experiments with realistic ML workloads, including a neural network trained for image classification. Our results show that Serene can speed up training by up to 40% in emulation scenarios by reducing drastically the cumulative waiting time compared to a synchronous baseline.

Förderung

  • Conselho Nacional de Desenvolvimento Científico e Tecnológico
  • Coordenação de Aperfeiçoamento de Pessoal de Nível Superior
  • Fundação de Amparo à Pesquisa do Estado de São Paulo

Zitieren

APA

Nunes, D. C., Coelho, B. L., Parizotto, R., & Schaeffer‐Filho, A. E. (2025). No Worker Left (Too Far) Behind: Dynamic Hybrid Synchronization for In‐Network ML Aggregation. International Journal of Network Management, 35(1). https://doi.org/10.1002/nem.2290

Vancouver

Nunes DC, Coelho BL, Parizotto R, Schaeffer‐Filho AE. No Worker Left (Too Far) Behind: Dynamic Hybrid Synchronization for In‐Network ML Aggregation. International Journal of Network Management. 2025;35(1). doi:10.1002/nem.2290

BibTeX
@article{Nunes2025,
  author = {Nunes, Diego Cardoso and Coelho, Bruno Loureiro and Parizotto, Ricardo and Schaeffer‐Filho, Alberto Egon},
  title = {No Worker Left (Too Far) Behind: Dynamic Hybrid Synchronization for In‐Network ML Aggregation},
  journal = {International Journal of Network Management},
  year = {2025},
  volume = {35},
  number = {1},
  doi = {10.1002/nem.2290}
}
Kostenlos starten →