A High-Parallelism Softmax Hardware-Software Co-Design for Fast and Efficient LLM Inference
ACM Transactions on Design Automation of Electronic Systems · 30.09.2026 · 31 (5) · 1-20
DOI: 10.1145/3801553 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Design Automation of Electronic Systems, online erschienen am 10.04.2026. Verfasst von Chenyi Wen und 5 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Gefördert von: NSFC, Zhejiang University Education Foundation Qizhen Scholar Foundation.
Abstract (Originalsprache)
Large language models (LLMs) have been the driving force behind significant advancements in artificial intelligence. However, their unique self-attention mechanism leads to difficulties in accelerating the inference. Softmax, with its complex nonlinear operations and low parallelism, significantly limits the efficiency of LLMs for long sequences. This work proposes a novel high-parallelism hardware/software co-design Softmax solution. By incorporating a sum estimation algorithm, we eliminate the need for complex exponential on all elements. A high-speed low-energy hardware architecture is introduced by applying high-parallelism statistical module and simplified division module. Experimental results demonstrate that our approach achieves a 9.90–44.75% reduction in latency and up to a 17.36% reduction in energy consumption compared with state-of-the-art Softmax hardware, with minimal impact on model inference accuracy.
Förderung
- NSFC
- Zhejiang University Education Foundation Qizhen Scholar Foundation
Zitieren
APA
Wen, C., Du, H., He, X., Yan, Z., Sun, Q., & Zhuo, C. (2026). A High-Parallelism Softmax Hardware-Software Co-Design for Fast and Efficient LLM Inference. ACM Transactions on Design Automation of Electronic Systems, 31(5), 1-20. https://doi.org/10.1145/3801553
Vancouver
Wen C, Du H, He X, Yan Z, Sun Q, Zhuo C. A High-Parallelism Softmax Hardware-Software Co-Design for Fast and Efficient LLM Inference. ACM Transactions on Design Automation of Electronic Systems. 2026;31(5):1-20. doi:10.1145/3801553
BibTeX
@article{Wen2026,
author = {Wen, Chenyi and Du, Haonan and He, Xuyang and Yan, Zheyu and Sun, Qi and Zhuo, Cheng},
title = {A High-Parallelism Softmax Hardware-Software Co-Design for Fast and Efficient LLM Inference},
journal = {ACM Transactions on Design Automation of Electronic Systems},
year = {2026},
volume = {31},
number = {5},
pages = {1--20},
doi = {10.1145/3801553}
}
EUVIMED