EUVIMEDEUVIMED entdecken
Crossref

HyGIN: Hybrid CPU/GPU-Initiated Communication for Mixture-of-Experts Training

Jinghan Huang, Nam Sung Kim

IEEE Computer Architecture Letters · 2026-07 · 25 (2) · 275-278

DOI: 10.1109/lca.2026.3711652 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Computer Architecture Letters, 2026-07. Verfasst von Jinghan Huang und 1 weiteren Autor:innen. Die Arbeit nennt 14 Literaturangaben. Gefördert von: PRISM, Semiconductor Research Corporation (SRC) program, IBM-Illinois Discovery Accelerator Institute.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

As large language models (LLMs) grow in size, they are trained on an increasing number of GPUs, making training time heavily tied to network communication. To scale LLMs further, recent models adopt Mixture-of-Experts (MoE) architectures, which make communication even more demanding and interleave small token messages with large activation and gradient messages. Such communications are traditionally issued by the CPU, which adds GPU kernel-launch and CPU–GPU synchronization overhead, motivating recent GPU-initiated communication in which GPU threads issue it directly. However, the practical benefit of GPU initiation remains unclear. We characterize CPU- versus GPU-initiated communication across message sizes and find that GPU initiation is faster at small sizes but slower at large sizes. Motivated by this characterization, we propose HyGIN, a message-size-aware hybrid that routes each network communication to either the CPU- or GPU-initiated path inside one persistent GPU kernel. In end-to-end simulation with Qwen3 235B and DeepSeek-V2 236B, HyGIN reduces training time by up to 11% and 8% over pure CPU- and GPU-initiated communication, respectively.

Förderung

  • PRISM
  • Semiconductor Research Corporation (SRC) program
  • IBM-Illinois Discovery Accelerator Institute

Zitieren

APA

Huang, J., & Kim, N. S. (2026). HyGIN: Hybrid CPU/GPU-Initiated Communication for Mixture-of-Experts Training. IEEE Computer Architecture Letters, 25(2), 275-278. https://doi.org/10.1109/lca.2026.3711652

Vancouver

Huang J, Kim NS. HyGIN: Hybrid CPU/GPU-Initiated Communication for Mixture-of-Experts Training. IEEE Computer Architecture Letters. 2026;25(2):275-278. doi:10.1109/lca.2026.3711652

BibTeX
@article{Huang2026,
  author = {Huang, Jinghan and Kim, Nam Sung},
  title = {HyGIN: Hybrid CPU/GPU-Initiated Communication for Mixture-of-Experts Training},
  journal = {IEEE Computer Architecture Letters},
  year = {2026},
  volume = {25},
  number = {2},
  pages = {275--278},
  doi = {10.1109/lca.2026.3711652}
}
Kostenlos starten →