Canvas+: Scalable Collective Communication Scheduling for Large-Scale GPU Clusters
IEEE Transactions on Networking · 2026 · 34 · 7965-7980
DOI: 10.1109/ton.2026.3747828 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Networking, 2026. Verfasst von Chenyang Hei und 10 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Liaoning Revitalization Talents Program.
Abstract (Originalsprache)
State-of-the-art deep learning models rely on large GPU clusters and various parallelism strategies, which in turn depend on collective communication (CC) operators to synchronize data. While vendor libraries (e.g., NCCL, RCCL) provide standard CC algorithms, they often suffer from bandwidth bottlenecks in modern, complex topologies. Recent synthesis-based methods improve performance but face four fundamental limitations: the combinatorial explosion of the scheduling space, monolithic execution prone to transient congestion, bandwidth waste from sequential micro-batches, and stranded inter-group bandwidth under strict closed-group routing in spatially multiplexed clusters. We propose Canvas+, a scalable and near-optimal CC scheduling framework that overcomes these architectural bottlenecks. Canvas+ introduces four key innovations: 1) Hierarchical synthesis with semantic subgraph stitching to decompose the global scheduling problem and accelerate synthesis by reusing verified schedules. 2) Collective decomposition with symmetry-induced load balancing to enable structured, multi-stage, and congestion-free algorithm generation. 3) Cross-micro-batch pipeline scheduling to eliminate sequential execution dependencies and maximize concurrent link utilization. 4) Fragmented resource integration via relay routing to reclaim stranded scale-up bandwidth by routing flows through idle, non-participating nodes. Extensive evaluations show that Canvas+ achieves up to $1.98\times $ and $3.56\times $ bandwidth speedups over TACCL and TE-CCL, respectively, while reducing synthesis time by 3–4 orders of magnitude at large scales. Relay-assisted routing further compresses scheduling epochs by up to 44.4% in spatially multiplexed clusters.
Förderung
- National Natural Science Foundation of China
- Liaoning Revitalization Talents Program
Zitieren
APA
Hei, C., Li, F., Zheng, D., Zhao, Y., Gao, C., Li, Z., Sha, X., Liu, T., Wan, H., Wang, X., & Cao, J. (2026). Canvas+: Scalable Collective Communication Scheduling for Large-Scale GPU Clusters. IEEE Transactions on Networking, 34, 7965-7980. https://doi.org/10.1109/ton.2026.3747828
Vancouver
Hei C, Li F, Zheng D, Zhao Y, Gao C, Li Z, et al. Canvas+: Scalable Collective Communication Scheduling for Large-Scale GPU Clusters. IEEE Transactions on Networking. 2026;34:7965-7980. doi:10.1109/ton.2026.3747828
BibTeX
@article{Hei2026,
author = {Hei, Chenyang and Li, Fuliang and Zheng, Donglin and Zhao, Yi and Gao, Chengxi and Li, Ziming and Sha, Xiuzhu and Liu, Tongrui and Wan, Haorui and Wang, Xingwei and Cao, Jiannong},
title = {Canvas+: Scalable Collective Communication Scheduling for Large-Scale GPU Clusters},
journal = {IEEE Transactions on Networking},
year = {2026},
volume = {34},
pages = {7965--7980},
doi = {10.1109/ton.2026.3747828}
}
EUVIMED