4.6-Bit Quantization for Fast and Accurate Neural Network Inference on CPUs
Mathematics · 23.02.2024 · 12 (5) · 651
DOI: 10.3390/math12050651 ↗Kurz gefasst
Fachartikel in der Zeitschrift Mathematics, online erschienen am 23.02.2024. Verfasst von Anton Trusov und 3 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Quantization is a widespread method for reducing the inference time of neural networks on mobile Central Processing Units (CPUs). Eight-bit quantized networks demonstrate similarly high quality as full precision models and perfectly fit the hardware architecture with one-byte coefficients and thirty-two-bit dot product accumulators. Lower precision quantizations usually suffer from noticeable quality loss and require specific computational algorithms to outperform eight-bit quantization. In this paper, we propose a novel 4.6-bit quantization scheme that allows for more efficient use of CPU resources. This scheme has more quantization bins than four-bit quantization and is more accurate while preserving the computational efficiency of the later (it runs only 4% slower). Our multiplication uses a combination of 16- and 32-bit accumulators and avoids multiplication depth limitation, which the previous 4-bit multiplication algorithm had. The experiments with different convolutional neural networks on CIFAR-10 and ImageNet datasets show that 4.6-bit quantized networks are 1.5–1.6 times faster than eight-bit networks on the ARMv8 CPU. Regarding the quality, the results of the 4.6-bit quantized network are close to the mean of four-bit and eight-bit networks of the same architecture. Therefore, 4.6-bit quantization may serve as an intermediate solution between fast and inaccurate low-bit network quantizations and accurate but relatively slow eight-bit ones.
Zitieren
APA
Trusov, A., Limonova, E., Nikolaev, D., & Arlazarov, V. V. (2024). 4.6-Bit Quantization for Fast and Accurate Neural Network Inference on CPUs. Mathematics, 12(5), 651. https://doi.org/10.3390/math12050651
Vancouver
Trusov A, Limonova E, Nikolaev D, Arlazarov VV. 4.6-Bit Quantization for Fast and Accurate Neural Network Inference on CPUs. Mathematics. 2024;12(5):651. doi:10.3390/math12050651
BibTeX
@article{Trusov2024,
author = {Trusov, Anton and Limonova, Elena and Nikolaev, Dmitry and Arlazarov, Vladimir V.},
title = {4.6-Bit Quantization for Fast and Accurate Neural Network Inference on CPUs},
journal = {Mathematics},
year = {2024},
volume = {12},
number = {5},
pages = {651},
doi = {10.3390/math12050651}
}
EUVIMED