Efficient Addition-Based Sparse GEMM for Fast Ternary Large Language Model Inference on Edge Devices
ACM Transactions on Embedded Computing Systems · 31.07.2026 · 25 (4) · 1-29
DOI: 10.1145/3807782 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Embedded Computing Systems, online erschienen am 17.06.2026. Verfasst von Shien Zhu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Large Language Models (LLMs) are the new dominant application but suffer from high memory and computational cost. Ternary LLMs have been proposed for easier deployment on edge platforms as 2-bit ternary weights {-1, 0, +1} can reduce the model size by 16× compared to 32-bit Floating-Point (FP32) representations. In addition, ternary General Matrix Multiplication (GEMM) can reduce the computational complexity by performing addition and subtraction operations with non-zero weights only. However, existing Central Processing Unit (CPU) and Graphic Processing Unit (GPU) do not support native 2-bit operations, and existing libraries like PyTorch and Compute Unified Device Architecture (CUDA) do not have dedicated computing kernels for ternary weights. Moreover, existing sparse formats like Compressed Sparse Column are not optimized for ternary values, causing extra storage and decompression overhead. In this article, we accelerate ternary LLMs on edge devices through efficient data formats and specialized computing kernels. We propose an efficient ternary sparse data format storing only the indices of non-zero values and simplifying the decompression at runtime. We also design a novel ternary GEMM algorithm that performs sparse addition on activations instead of multiplication to reduce the computation complexity. It achieves a 4× theoretical speedup over dense GEMM with 50% sparsity in weights. We have implemented these algorithms and optimized computing kernels on both x86 CPU and Nvidia GPUs. Evaluation results show that they achieve 1.3-3.9× speedup over Eigen Sparse GEMM, 3.3-6.9× speedup over PyTorch Sparse GEMM, and around 5.5× speedup over cuSPARSE. The GPU implementation can serve Llama-3 3B and 8B models on an RTX-3080Ti with 22 and 7 tokens/s, while the full-precision versions run out of memory.
Zitieren
APA
Zhu, S., Fu, G., Kjoseva, M., & Alonso, G. (2026). Efficient Addition-Based Sparse GEMM for Fast Ternary Large Language Model Inference on Edge Devices. ACM Transactions on Embedded Computing Systems, 25(4), 1-29. https://doi.org/10.1145/3807782
Vancouver
Zhu S, Fu G, Kjoseva M, Alonso G. Efficient Addition-Based Sparse GEMM for Fast Ternary Large Language Model Inference on Edge Devices. ACM Transactions on Embedded Computing Systems. 2026;25(4):1-29. doi:10.1145/3807782
BibTeX
@article{Zhu2026,
author = {Zhu, Shien and Fu, Guanshujie and Kjoseva, Mila and Alonso, Gustavo},
title = {Efficient Addition-Based Sparse GEMM for Fast Ternary Large Language Model Inference on Edge Devices},
journal = {ACM Transactions on Embedded Computing Systems},
year = {2026},
volume = {25},
number = {4},
pages = {1--29},
doi = {10.1145/3807782}
}
EUVIMED