EUVIMEDEUVIMED entdecken
Crossref

FLRQ: Faster LLM Quantization with Flexible Low-Rank Matrix Sketching

Hongyaoxing Gu, Lijuan Hu, Shuzi Niu, Fangfang Liu

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (26) · 21369-21377

DOI: 10.1609/aaai.v40i26.39283 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Hongyaoxing Gu und 3 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Traditional post-training quantization (PTQ) is considered an effective approach to reduce model size and accelerate inference of large-scale language models (LLMs). However, existing low-rank PTQ methods require costly fine-tuning to determine a compromise rank for diverse data and layers in large models, failing to exploit their full potential. Additionally, the current SVD-based low-rank approximation compounds the computational overhead. In this work, we thoroughly analyze the varying effectiveness of low-rank approximation across different layers in representative models. Accordingly, we introduce Flexible Low-Rank Quantization (FLRQ), a novel solution designed to quickly identify the accuracy-optimal ranks and aggregate them to achieve minimal storage combinations. FLRQ comprises two powerful components, Rank1-Sketch-based Flexible Rank Selection (R1-FLR) and Best Low-rank Approximation under Clipping (BLC). R1-FLR applies the R1-Sketch with Gaussian projection for the fast low-rank approximation, enabling outlier-aware rank extraction for each layer. Meanwhile, BLC aims at minimizing the low-rank quantization error under the scaling and clipping strategy through an iterative method. FLRQ demonstrates strong effectiveness and robustness in comprehensive experiments, achieving state-of-the-art performance in both quantization quality and algorithm efficiency.

Zitieren

APA

Gu, H., Hu, L., Niu, S., & Liu, F. (2026). FLRQ: Faster LLM Quantization with Flexible Low-Rank Matrix Sketching. Proceedings of the AAAI Conference on Artificial Intelligence, 40(26), 21369-21377. https://doi.org/10.1609/aaai.v40i26.39283

Vancouver

Gu H, Hu L, Niu S, Liu F. FLRQ: Faster LLM Quantization with Flexible Low-Rank Matrix Sketching. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(26):21369-21377. doi:10.1609/aaai.v40i26.39283

BibTeX
@article{Gu2026,
  author = {Gu, Hongyaoxing and Hu, Lijuan and Niu, Shuzi and Liu, Fangfang},
  title = {FLRQ: Faster LLM Quantization with Flexible Low-Rank Matrix Sketching},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {26},
  pages = {21369--21377},
  doi = {10.1609/aaai.v40i26.39283}
}
Kostenlos starten →