EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

Attention Distribution-Aware Softmax for NPU-Accelerated On-Device Inference of LLMs: An Edge-Oriented Approximation Design

Sanoop Sadheerthan, Min-Jie Hsu, Chih-Hsiang Huang, Yin-Tien Wang

Electronics · 2026-03-20 · 15 (6) · 1312

DOI: 10.3390/electronics15061312 ↗

At a glance

Journal article in Electronics, published online on 2026-03-20. Written by Sanoop Sadheerthan and 3 co-authors. Cited 2 times so far (Crossref); 42 references. The full text is freely available (open access, CC BY 4.0). Funded by: Innodisk Corporation, National Taiwan Normal University and the Ministry of Education (MOE), Taiwan, National Science and Technology Council (NSTC), Taiwan.

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Low-power NPUs enable on-device LLM inference through efficient integer and fixed-point algebra, yet their lack of native exponential support makes Transformer softmax a critical performance bottleneck. Existing NPU kernels approximate ex using uniform piecewise polynomials to enable O(1) SIMD indexing, but this wastes computation by applying high-degree arithmetic indiscriminately in every segment. Conversely, fully adaptive approaches maximize statistical fidelity but introduce pipeline stalls due to comparator-based boundary search. To bridge this gap, we propose an attention distribution-aware softmax that uses Particle Swarm Optimization (PSO) to define non-uniform segments and variable polynomial degrees, prioritizing finer granularity and lower arithmetic complexity in attention-dense regions. To ensure efficiency, we snap boundaries into a 128-bin LUT, enabling O(1) retrieval of segment parameters without branching. Inference measurements show that this favors low-degree execution, minimizing exp-kernel overhead. Using TinyLlama-1.1B-Chat as a testbed, the proposed weighted design reduces cycles per call exp kernel (CPC) by 18.5% versus an equidistant uniform Degree-4 baseline and 13.1% versus uniform Degree-3, while preserving ranking fidelity. These results show that grid-snapped, variable-degree approximation can improve softmax efficiency while largely preserving attention ranking fidelity, enabling accurate edge LLM inference.

Funding

  • Innodisk Corporation
  • National Taiwan Normal University and the Ministry of Education (MOE), Taiwan
  • National Science and Technology Council (NSTC), Taiwan

Cite

APA

Sadheerthan, S., Hsu, M., Huang, C., & Wang, Y. (2026). Attention Distribution-Aware Softmax for NPU-Accelerated On-Device Inference of LLMs: An Edge-Oriented Approximation Design. Electronics, 15(6), 1312. https://doi.org/10.3390/electronics15061312

Vancouver

Sadheerthan S, Hsu M, Huang C, Wang Y. Attention Distribution-Aware Softmax for NPU-Accelerated On-Device Inference of LLMs: An Edge-Oriented Approximation Design. Electronics. 2026;15(6):1312. doi:10.3390/electronics15061312

BibTeX
@article{Sadheerthan2026,
  author = {Sadheerthan, Sanoop and Hsu, Min-Jie and Huang, Chih-Hsiang and Wang, Yin-Tien},
  title = {Attention Distribution-Aware Softmax for NPU-Accelerated On-Device Inference of LLMs: An Edge-Oriented Approximation Design},
  journal = {Electronics},
  year = {2026},
  volume = {15},
  number = {6},
  pages = {1312},
  doi = {10.3390/electronics15061312}
}
Start for free →