EUVIMEDEUVIMED entdecken
Crossref

Canvas+: Scalable Collective Communication Scheduling for Large-Scale GPU Clusters

Chenyang Hei, Fuliang Li, Donglin Zheng, Yi Zhao, Chengxi Gao, Ziming Li, Xiuzhu Sha, Tongrui Liu, Haorui Wan, Xingwei Wang, Jiannong Cao

IEEE Transactions on Networking · 2026 · 34 · 7965-7980

DOI: 10.1109/ton.2026.3747828 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Networking, 2026. Verfasst von Chenyang Hei und 10 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Liaoning Revitalization Talents Program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

State-of-the-art deep learning models rely on large GPU clusters and various parallelism strategies, which in turn depend on collective communication (CC) operators to synchronize data. While vendor libraries (e.g., NCCL, RCCL) provide standard CC algorithms, they often suffer from bandwidth bottlenecks in modern, complex topologies. Recent synthesis-based methods improve performance but face four fundamental limitations: the combinatorial explosion of the scheduling space, monolithic execution prone to transient congestion, bandwidth waste from sequential micro-batches, and stranded inter-group bandwidth under strict closed-group routing in spatially multiplexed clusters. We propose Canvas+, a scalable and near-optimal CC scheduling framework that overcomes these architectural bottlenecks. Canvas+ introduces four key innovations: 1) Hierarchical synthesis with semantic subgraph stitching to decompose the global scheduling problem and accelerate synthesis by reusing verified schedules. 2) Collective decomposition with symmetry-induced load balancing to enable structured, multi-stage, and congestion-free algorithm generation. 3) Cross-micro-batch pipeline scheduling to eliminate sequential execution dependencies and maximize concurrent link utilization. 4) Fragmented resource integration via relay routing to reclaim stranded scale-up bandwidth by routing flows through idle, non-participating nodes. Extensive evaluations show that Canvas+ achieves up to $1.98\times $ and $3.56\times $ bandwidth speedups over TACCL and TE-CCL, respectively, while reducing synthesis time by 3–4 orders of magnitude at large scales. Relay-assisted routing further compresses scheduling epochs by up to 44.4% in spatially multiplexed clusters.

Förderung

  • National Natural Science Foundation of China
  • Liaoning Revitalization Talents Program

Zitieren

APA

Hei, C., Li, F., Zheng, D., Zhao, Y., Gao, C., Li, Z., Sha, X., Liu, T., Wan, H., Wang, X., & Cao, J. (2026). Canvas+: Scalable Collective Communication Scheduling for Large-Scale GPU Clusters. IEEE Transactions on Networking, 34, 7965-7980. https://doi.org/10.1109/ton.2026.3747828

Vancouver

Hei C, Li F, Zheng D, Zhao Y, Gao C, Li Z, et al. Canvas+: Scalable Collective Communication Scheduling for Large-Scale GPU Clusters. IEEE Transactions on Networking. 2026;34:7965-7980. doi:10.1109/ton.2026.3747828

BibTeX
@article{Hei2026,
  author = {Hei, Chenyang and Li, Fuliang and Zheng, Donglin and Zhao, Yi and Gao, Chengxi and Li, Ziming and Sha, Xiuzhu and Liu, Tongrui and Wan, Haorui and Wang, Xingwei and Cao, Jiannong},
  title = {Canvas+: Scalable Collective Communication Scheduling for Large-Scale GPU Clusters},
  journal = {IEEE Transactions on Networking},
  year = {2026},
  volume = {34},
  pages = {7965--7980},
  doi = {10.1109/ton.2026.3747828}
}
Kostenlos starten →