EUVIMEDEUVIMED entdecken
Crossref

PRC: Process-centric Rate Control for Transient Stragglers in Hybrid-Parallel Training

Taekyoung Han, Daewook Hwang, Shinik Park, Donggyu Yang, Gibum Park, Sanghyun Han, Jaeyoon Hwang, Byunggu Kang, Sangtae Ha, Kyunghan Lee

Proceedings of the ACM on Networking · 01.06.2026 · 4 (CoNEXT2) · 1-23

DOI: 10.1145/3808669 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Networking, online erschienen am 01.06.2026. Verfasst von Taekyoung Han und 9 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 83 Literaturangaben. Gefördert von: Institute of Information & Communications Technology Planning & Evaluation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Stragglers remain an open challenge in large model training, forcing other GPUs to wait at synchronization barriers. While severe, persistent stragglers have been actively studied, their transient counterparts are often overlooked due to their short-lived nature. We reveal that the interplay between the transient straggler effects and network contention drastically amplifies the relative delays across workers, significantly slowing down training iterations. To address this, an intuitive approach is to prioritize bandwidth for the straggler, allowing it to catch up to the front-runners. However, while the transport layer is well-positioned to accommodate such responsive bandwidth control, conventional designs lack visibility into process-level progress. To bridge this gap, we propose PRC (Process-centric Rate Control), a new sending-rate control designed to mitigate the transient straggler effects. PRC adjusts NIC sending rates by inferring local GPU process-level information, enabling transient straggler processes to utilize more bandwidth on time. Extensive experiments on both a real-world cluster and large-scale simulations confirm that PRC effectively accelerates transient stragglers, achieving a training speedup of up to 28% compared to using state-of-the-art datacenter congestion control schemes.

Förderung

  • Institute of Information & Communications Technology Planning & Evaluation

Zitieren

APA

Han, T., Hwang, D., Park, S., Yang, D., Park, G., Han, S., Hwang, J., Kang, B., Ha, S., & Lee, K. (2026). PRC: Process-centric Rate Control for Transient Stragglers in Hybrid-Parallel Training. Proceedings of the ACM on Networking, 4(CoNEXT2), 1-23. https://doi.org/10.1145/3808669

Vancouver

Han T, Hwang D, Park S, Yang D, Park G, Han S, et al. PRC: Process-centric Rate Control for Transient Stragglers in Hybrid-Parallel Training. Proceedings of the ACM on Networking. 2026;4(CoNEXT2):1-23. doi:10.1145/3808669

BibTeX
@article{Han2026,
  author = {Han, Taekyoung and Hwang, Daewook and Park, Shinik and Yang, Donggyu and Park, Gibum and Han, Sanghyun and Hwang, Jaeyoon and Kang, Byunggu and Ha, Sangtae and Lee, Kyunghan},
  title = {PRC: Process-centric Rate Control for Transient Stragglers in Hybrid-Parallel Training},
  journal = {Proceedings of the ACM on Networking},
  year = {2026},
  volume = {4},
  number = {CoNEXT2},
  pages = {1--23},
  doi = {10.1145/3808669}
}
Kostenlos starten →