HAGC: A Hardware-Aware Gradient Compression framework for distributed deep learning
Journal of Systems Architecture · 2026-06 · 175 · 103770
DOI: 10.1016/j.sysarc.2026.103770 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Systems Architecture, 2026-06. Verfasst von Aiqiang Yang und 6 weiteren Autor:innen. Die Arbeit nennt 47 Literaturangaben. Gefördert von: Ministry of Science and Technology of the People's Republic of China, National Key Research and Development Program of China.
Abstract (Originalsprache)
Distributed deep learning faces a persistent communication bottleneck as models scale exponentially. Standard software-based compression efforts often fail to exploit specialized GPU units, resulting in a computational lag that offsets intended bandwidth gains. This paper introduces Hardware-Aware Gradient Compression (HAGC), a framework that offloads the entire compression workload to hardware-native operations. Restructuring the Bilateral Hadamard Transform from memory-bound recursion into dense matrix operations allows HAGC to execute directly on Tensor Core mixed-precision units. This structural synchronization masks compression latency, while an integrated error feedback mechanism ensures convergence stability. The framework supports a systematic co-design strategy that calibrates training parameters to align processor capacity with training stability. Empirical evaluations on NVIDIA A100 GPUs demonstrate substantial performance gains, achieving up to a 3.15 × end-to-end speedup for data-intensive models such as VGG-19. The resulting framework reduces energy consumption by 2.9 × and memory footprint by nearly 4 × , while maintaining model fidelity comparable to full-precision baselines. Such findings validate that bridging algorithmic design with physical hardware capabilities is essential for building sustainable, high-performance distributed systems.
Förderung
- Ministry of Science and Technology of the People's Republic of China
- National Key Research and Development Program of China
Zitieren
APA
Yang, A., Liu, J., Yang, B., Zhang, X., Wang, Q., Mo, Z., & Li, K. (2026). HAGC: A Hardware-Aware Gradient Compression framework for distributed deep learning. Journal of Systems Architecture, 175, 103770. https://doi.org/10.1016/j.sysarc.2026.103770
Vancouver
Yang A, Liu J, Yang B, Zhang X, Wang Q, Mo Z, et al. HAGC: A Hardware-Aware Gradient Compression framework for distributed deep learning. Journal of Systems Architecture. 2026;175:103770. doi:10.1016/j.sysarc.2026.103770
BibTeX
@article{Yang2026,
author = {Yang, Aiqiang and Liu, Jie and Yang, Bo and Zhang, Xiang and Wang, Qinglin and Mo, Zeyao and Li, Keqin},
title = {HAGC: A Hardware-Aware Gradient Compression framework for distributed deep learning},
journal = {Journal of Systems Architecture},
year = {2026},
volume = {175},
pages = {103770},
doi = {10.1016/j.sysarc.2026.103770}
}
EUVIMED