EUVIMEDEUVIMED entdecken
Crossref

A High-Parallelism Softmax Hardware-Software Co-Design for Fast and Efficient LLM Inference

Chenyi Wen, Haonan Du, Xuyang He, Zheyu Yan, Qi Sun, Cheng Zhuo

ACM Transactions on Design Automation of Electronic Systems · 30.09.2026 · 31 (5) · 1-20

DOI: 10.1145/3801553 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Design Automation of Electronic Systems, online erschienen am 10.04.2026. Verfasst von Chenyi Wen und 5 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Gefördert von: NSFC, Zhejiang University Education Foundation Qizhen Scholar Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) have been the driving force behind significant advancements in artificial intelligence. However, their unique self-attention mechanism leads to difficulties in accelerating the inference. Softmax, with its complex nonlinear operations and low parallelism, significantly limits the efficiency of LLMs for long sequences. This work proposes a novel high-parallelism hardware/software co-design Softmax solution. By incorporating a sum estimation algorithm, we eliminate the need for complex exponential on all elements. A high-speed low-energy hardware architecture is introduced by applying high-parallelism statistical module and simplified division module. Experimental results demonstrate that our approach achieves a 9.90–44.75% reduction in latency and up to a 17.36% reduction in energy consumption compared with state-of-the-art Softmax hardware, with minimal impact on model inference accuracy.

Förderung

  • NSFC
  • Zhejiang University Education Foundation Qizhen Scholar Foundation

Zitieren

APA

Wen, C., Du, H., He, X., Yan, Z., Sun, Q., & Zhuo, C. (2026). A High-Parallelism Softmax Hardware-Software Co-Design for Fast and Efficient LLM Inference. ACM Transactions on Design Automation of Electronic Systems, 31(5), 1-20. https://doi.org/10.1145/3801553

Vancouver

Wen C, Du H, He X, Yan Z, Sun Q, Zhuo C. A High-Parallelism Softmax Hardware-Software Co-Design for Fast and Efficient LLM Inference. ACM Transactions on Design Automation of Electronic Systems. 2026;31(5):1-20. doi:10.1145/3801553

BibTeX
@article{Wen2026,
  author = {Wen, Chenyi and Du, Haonan and He, Xuyang and Yan, Zheyu and Sun, Qi and Zhuo, Cheng},
  title = {A High-Parallelism Softmax Hardware-Software Co-Design for Fast and Efficient LLM Inference},
  journal = {ACM Transactions on Design Automation of Electronic Systems},
  year = {2026},
  volume = {31},
  number = {5},
  pages = {1--20},
  doi = {10.1145/3801553}
}
Kostenlos starten →