Spine-Free Networks for Large Language Model Training
IEEE Micro · 2025-03 · 45 (2) · 18-25
DOI: 10.1109/mm.2025.3540663 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Micro, 2025-03. Verfasst von Weiyang Wang und 1 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 12 Literaturangaben. Gefördert von: Defense Advanced Research Projects Agency, National Science Foundation.
Abstract (Originalsprache)
We study the optimal parallelization strategy of large language models (LLMs) and demonstrate that LLM training workloads generate sparse communication patterns in the network. Consequently, we argue that LLM training clusters do not require any-to-any full-bisection networks. We then propose Rail-only, a novel datacenter network architecture tailored to LLMs’ unique communication patterns. Rail-only networks eliminate the spine layer of conventional fabrics, resulting in lower cost and energy consumption. We demonstrate that Rail-only networks achieve the same training performance while reducing network cost by 38% to 77% and network power consumption by 37% to 75%, compared to traditional GPU clusters with full-bisection bandwidth.
Förderung
- Defense Advanced Research Projects Agency
- National Science Foundation
Zitieren
APA
Wang, W., & Ghobadi, M. (2025). Spine-Free Networks for Large Language Model Training. IEEE Micro, 45(2), 18-25. https://doi.org/10.1109/mm.2025.3540663
Vancouver
Wang W, Ghobadi M. Spine-Free Networks for Large Language Model Training. IEEE Micro. 2025;45(2):18-25. doi:10.1109/mm.2025.3540663
BibTeX
@article{Wang2025,
author = {Wang, Weiyang and Ghobadi, Manya},
title = {Spine-Free Networks for Large Language Model Training},
journal = {IEEE Micro},
year = {2025},
volume = {45},
number = {2},
pages = {18--25},
doi = {10.1109/mm.2025.3540663}
}
EUVIMED