Neural Network Driven Quantization Aware Optimization for Low Latency Large Language Model Inference
International Journal of Scientific Research in Computer Science, Engineering and Information Technology · 30.06.2024 · 10 (3) · 1162-1170
DOI: 10.32628/cseit25113584 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Scientific Research in Computer Science, Engineering and Information Technology, online erschienen am 30.06.2024. Verfasst von Navya Veginati. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large Language Models (LLMs) have proven to be remarkably successful in natural language processing applications, but the cost of computation, large memory consumption and longer inference time represent limiting factors in their use. To overcome these challenges, this paper suggests a Neural Network Driven Quantization Aware Optimization framework. The approach integrates the concept of training with the understanding of quantization with a neural controller that dynamically selects the most effective quantification levels to apply in the different layers of the model. Adaptive quantization and latency-sensitive optimization can be employed to attain significantly reduced inference time and memory footprint without loss of model accuracy with the proposed technique. The experimental analysis demonstrates that the proposed framework is more effective and works better compared to the existing methods which incorporate GPTQ, LLM-MQ, and SVD-LLM. The results indicate the applicability of smart, learning-based optimization techniques to support real-time and resource-easy implementation of LLM.
Zitieren
APA
Veginati, N. (2024). Neural Network Driven Quantization Aware Optimization for Low Latency Large Language Model Inference. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 10(3), 1162-1170. https://doi.org/10.32628/cseit25113584
Vancouver
Veginati N. Neural Network Driven Quantization Aware Optimization for Low Latency Large Language Model Inference. International Journal of Scientific Research in Computer Science, Engineering and Information Technology. 2024;10(3):1162-1170. doi:10.32628/cseit25113584
BibTeX
@article{Veginati2024,
author = {Veginati, Navya},
title = {Neural Network Driven Quantization Aware Optimization for Low Latency Large Language Model Inference},
journal = {International Journal of Scientific Research in Computer Science, Engineering and Information Technology},
year = {2024},
volume = {10},
number = {3},
pages = {1162--1170},
doi = {10.32628/cseit25113584}
}
EUVIMED