EUVIMEDEUVIMED entdecken
Crossref

Understanding and Characterizing Communication Characteristics for Distributed Transformer Models

Quentin Anthony, Benjamin Michalowicz, Jacob Hatef, Lang Xu, Mustafa Abduljabbar, Aamir Shafi, Hari Subramoni, Dhabaleswar K. DK Panda

IEEE Micro · 2025-03 · 45 (2) · 8-17

DOI: 10.1109/mm.2025.3531323 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Micro, 2025-03. Verfasst von Quentin Anthony und 7 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 10 Literaturangaben. Gefördert von: National Science Foundation, XSEDE Resource Allocation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The transformer architecture has revolutionized many applications, such as large language models. This progress has been largely enabled by distributed training, yet communication remains a significant bottleneck. This article examines the communication behavior of transformer models, focusing on how different parallelism schemes in multinode/multi-GPU training communicate data. We use Generative Pre-trained Transformer-based language models as a case study due to their prevalence. We validate our empirical results using analytical models. Our analysis reveals practical insights and potential areas for further optimization in framework and high-performance computing middleware design.

Förderung

  • National Science Foundation
  • XSEDE Resource Allocation

Zitieren

APA

Anthony, Q., Michalowicz, B., Hatef, J., Xu, L., Abduljabbar, M., Shafi, A., Subramoni, H., & Panda, D. K. D. (2025). Understanding and Characterizing Communication Characteristics for Distributed Transformer Models. IEEE Micro, 45(2), 8-17. https://doi.org/10.1109/mm.2025.3531323

Vancouver

Anthony Q, Michalowicz B, Hatef J, Xu L, Abduljabbar M, Shafi A, et al. Understanding and Characterizing Communication Characteristics for Distributed Transformer Models. IEEE Micro. 2025;45(2):8-17. doi:10.1109/mm.2025.3531323

BibTeX
@article{Anthony2025,
  author = {Anthony, Quentin and Michalowicz, Benjamin and Hatef, Jacob and Xu, Lang and Abduljabbar, Mustafa and Shafi, Aamir and Subramoni, Hari and Panda, Dhabaleswar K. DK},
  title = {Understanding and Characterizing Communication Characteristics for Distributed Transformer Models},
  journal = {IEEE Micro},
  year = {2025},
  volume = {45},
  number = {2},
  pages = {8--17},
  doi = {10.1109/mm.2025.3531323}
}
Kostenlos starten →