FLRQ: Faster LLM Quantization with Flexible Low-Rank Matrix Sketching
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (26) · 21369-21377
DOI: 10.1609/aaai.v40i26.39283 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Hongyaoxing Gu und 3 weiteren Autor:innen.
Abstract (Originalsprache)
Traditional post-training quantization (PTQ) is considered an effective approach to reduce model size and accelerate inference of large-scale language models (LLMs). However, existing low-rank PTQ methods require costly fine-tuning to determine a compromise rank for diverse data and layers in large models, failing to exploit their full potential. Additionally, the current SVD-based low-rank approximation compounds the computational overhead. In this work, we thoroughly analyze the varying effectiveness of low-rank approximation across different layers in representative models. Accordingly, we introduce Flexible Low-Rank Quantization (FLRQ), a novel solution designed to quickly identify the accuracy-optimal ranks and aggregate them to achieve minimal storage combinations. FLRQ comprises two powerful components, Rank1-Sketch-based Flexible Rank Selection (R1-FLR) and Best Low-rank Approximation under Clipping (BLC). R1-FLR applies the R1-Sketch with Gaussian projection for the fast low-rank approximation, enabling outlier-aware rank extraction for each layer. Meanwhile, BLC aims at minimizing the low-rank quantization error under the scaling and clipping strategy through an iterative method. FLRQ demonstrates strong effectiveness and robustness in comprehensive experiments, achieving state-of-the-art performance in both quantization quality and algorithm efficiency.
Zitieren
APA
Gu, H., Hu, L., Niu, S., & Liu, F. (2026). FLRQ: Faster LLM Quantization with Flexible Low-Rank Matrix Sketching. Proceedings of the AAAI Conference on Artificial Intelligence, 40(26), 21369-21377. https://doi.org/10.1609/aaai.v40i26.39283
Vancouver
Gu H, Hu L, Niu S, Liu F. FLRQ: Faster LLM Quantization with Flexible Low-Rank Matrix Sketching. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(26):21369-21377. doi:10.1609/aaai.v40i26.39283
BibTeX
@article{Gu2026,
author = {Gu, Hongyaoxing and Hu, Lijuan and Niu, Shuzi and Liu, Fangfang},
title = {FLRQ: Faster LLM Quantization with Flexible Low-Rank Matrix Sketching},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {26},
pages = {21369--21377},
doi = {10.1609/aaai.v40i26.39283}
}
EUVIMED