HyGIN: Hybrid CPU/GPU-Initiated Communication for Mixture-of-Experts Training
IEEE Computer Architecture Letters · 2026-07 · 25 (2) · 275-278
DOI: 10.1109/lca.2026.3711652 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Computer Architecture Letters, 2026-07. Verfasst von Jinghan Huang und 1 weiteren Autor:innen. Die Arbeit nennt 14 Literaturangaben. Gefördert von: PRISM, Semiconductor Research Corporation (SRC) program, IBM-Illinois Discovery Accelerator Institute.
Abstract (Originalsprache)
As large language models (LLMs) grow in size, they are trained on an increasing number of GPUs, making training time heavily tied to network communication. To scale LLMs further, recent models adopt Mixture-of-Experts (MoE) architectures, which make communication even more demanding and interleave small token messages with large activation and gradient messages. Such communications are traditionally issued by the CPU, which adds GPU kernel-launch and CPU–GPU synchronization overhead, motivating recent GPU-initiated communication in which GPU threads issue it directly. However, the practical benefit of GPU initiation remains unclear. We characterize CPU- versus GPU-initiated communication across message sizes and find that GPU initiation is faster at small sizes but slower at large sizes. Motivated by this characterization, we propose HyGIN, a message-size-aware hybrid that routes each network communication to either the CPU- or GPU-initiated path inside one persistent GPU kernel. In end-to-end simulation with Qwen3 235B and DeepSeek-V2 236B, HyGIN reduces training time by up to 11% and 8% over pure CPU- and GPU-initiated communication, respectively.
Förderung
- PRISM
- Semiconductor Research Corporation (SRC) program
- IBM-Illinois Discovery Accelerator Institute
Zitieren
APA
Huang, J., & Kim, N. S. (2026). HyGIN: Hybrid CPU/GPU-Initiated Communication for Mixture-of-Experts Training. IEEE Computer Architecture Letters, 25(2), 275-278. https://doi.org/10.1109/lca.2026.3711652
Vancouver
Huang J, Kim NS. HyGIN: Hybrid CPU/GPU-Initiated Communication for Mixture-of-Experts Training. IEEE Computer Architecture Letters. 2026;25(2):275-278. doi:10.1109/lca.2026.3711652
BibTeX
@article{Huang2026,
author = {Huang, Jinghan and Kim, Nam Sung},
title = {HyGIN: Hybrid CPU/GPU-Initiated Communication for Mixture-of-Experts Training},
journal = {IEEE Computer Architecture Letters},
year = {2026},
volume = {25},
number = {2},
pages = {275--278},
doi = {10.1109/lca.2026.3711652}
}
EUVIMED