Attention Distribution-Aware Softmax for NPU-Accelerated On-Device Inference of LLMs: An Edge-Oriented Approximation Design
Electronics · 20.03.2026 · 15 (6) · 1312
DOI: 10.3390/electronics15061312 ↗Kurz gefasst
Fachartikel in der Zeitschrift Electronics, online erschienen am 20.03.2026. Verfasst von Sanoop Sadheerthan und 3 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Innodisk Corporation, National Taiwan Normal University and the Ministry of Education (MOE), Taiwan, National Science and Technology Council (NSTC), Taiwan.
Abstract (Originalsprache)
Low-power NPUs enable on-device LLM inference through efficient integer and fixed-point algebra, yet their lack of native exponential support makes Transformer softmax a critical performance bottleneck. Existing NPU kernels approximate ex using uniform piecewise polynomials to enable O(1) SIMD indexing, but this wastes computation by applying high-degree arithmetic indiscriminately in every segment. Conversely, fully adaptive approaches maximize statistical fidelity but introduce pipeline stalls due to comparator-based boundary search. To bridge this gap, we propose an attention distribution-aware softmax that uses Particle Swarm Optimization (PSO) to define non-uniform segments and variable polynomial degrees, prioritizing finer granularity and lower arithmetic complexity in attention-dense regions. To ensure efficiency, we snap boundaries into a 128-bin LUT, enabling O(1) retrieval of segment parameters without branching. Inference measurements show that this favors low-degree execution, minimizing exp-kernel overhead. Using TinyLlama-1.1B-Chat as a testbed, the proposed weighted design reduces cycles per call exp kernel (CPC) by 18.5% versus an equidistant uniform Degree-4 baseline and 13.1% versus uniform Degree-3, while preserving ranking fidelity. These results show that grid-snapped, variable-degree approximation can improve softmax efficiency while largely preserving attention ranking fidelity, enabling accurate edge LLM inference.
Förderung
- Innodisk Corporation
- National Taiwan Normal University and the Ministry of Education (MOE), Taiwan
- National Science and Technology Council (NSTC), Taiwan
Zitieren
APA
Sadheerthan, S., Hsu, M., Huang, C., & Wang, Y. (2026). Attention Distribution-Aware Softmax for NPU-Accelerated On-Device Inference of LLMs: An Edge-Oriented Approximation Design. Electronics, 15(6), 1312. https://doi.org/10.3390/electronics15061312
Vancouver
Sadheerthan S, Hsu M, Huang C, Wang Y. Attention Distribution-Aware Softmax for NPU-Accelerated On-Device Inference of LLMs: An Edge-Oriented Approximation Design. Electronics. 2026;15(6):1312. doi:10.3390/electronics15061312
BibTeX
@article{Sadheerthan2026,
author = {Sadheerthan, Sanoop and Hsu, Min-Jie and Huang, Chih-Hsiang and Wang, Yin-Tien},
title = {Attention Distribution-Aware Softmax for NPU-Accelerated On-Device Inference of LLMs: An Edge-Oriented Approximation Design},
journal = {Electronics},
year = {2026},
volume = {15},
number = {6},
pages = {1312},
doi = {10.3390/electronics15061312}
}
EUVIMED