Understanding and Characterizing Communication Characteristics for Distributed Transformer Models
IEEE Micro · 2025-03 · 45 (2) · 8-17
DOI: 10.1109/mm.2025.3531323 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Micro, 2025-03. Verfasst von Quentin Anthony und 7 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 10 Literaturangaben. Gefördert von: National Science Foundation, XSEDE Resource Allocation.
Abstract (Originalsprache)
The transformer architecture has revolutionized many applications, such as large language models. This progress has been largely enabled by distributed training, yet communication remains a significant bottleneck. This article examines the communication behavior of transformer models, focusing on how different parallelism schemes in multinode/multi-GPU training communicate data. We use Generative Pre-trained Transformer-based language models as a case study due to their prevalence. We validate our empirical results using analytical models. Our analysis reveals practical insights and potential areas for further optimization in framework and high-performance computing middleware design.
Förderung
- National Science Foundation
- XSEDE Resource Allocation
Zitieren
APA
Anthony, Q., Michalowicz, B., Hatef, J., Xu, L., Abduljabbar, M., Shafi, A., Subramoni, H., & Panda, D. K. D. (2025). Understanding and Characterizing Communication Characteristics for Distributed Transformer Models. IEEE Micro, 45(2), 8-17. https://doi.org/10.1109/mm.2025.3531323
Vancouver
Anthony Q, Michalowicz B, Hatef J, Xu L, Abduljabbar M, Shafi A, et al. Understanding and Characterizing Communication Characteristics for Distributed Transformer Models. IEEE Micro. 2025;45(2):8-17. doi:10.1109/mm.2025.3531323
BibTeX
@article{Anthony2025,
author = {Anthony, Quentin and Michalowicz, Benjamin and Hatef, Jacob and Xu, Lang and Abduljabbar, Mustafa and Shafi, Aamir and Subramoni, Hari and Panda, Dhabaleswar K. DK},
title = {Understanding and Characterizing Communication Characteristics for Distributed Transformer Models},
journal = {IEEE Micro},
year = {2025},
volume = {45},
number = {2},
pages = {8--17},
doi = {10.1109/mm.2025.3531323}
}
EUVIMED