Q-ROAR: Outlier-Aware Rescaling for RoPE Position Interpolation in Quantized Long-Context LLMs (Student Abstract)
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (48) · 41359-41361
DOI: 10.1609/aaai.v40i48.42269 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Ye Qiao und 1 weiteren Autor:innen.
Abstract (Originalsprache)
Extending LLM context windows is key for long-range tasks. RoPE-based position interpolation (PI) scales input length without retraining, and post-training quantization (PTQ) enables efficient deployment; however, combining PI with PTQ degrades accuracy due to long-context aliasing, dynamic-range dilation, axis-grid anisotropy, and outlier shifts that induce position-dependent logit noise. We give the first systematic analysis of PI+PTQ and propose two diagnostics: Interpolation Pressure (per-band phase-scaling sensitivity) and Tail Inflation Ratio (outlier shift from short to long contexts). We then introduce Q-ROAR, a RoPE-aware, weight-only stabilization that bands RoPE dimensions and lightly searches per-band scales for W_Q,W_K, with an optional symmetric variant. Q-ROAR needs only a tiny long-context dev set and no fine-tuning or kernel changes, recovering up to 0.7% accuracy and more than 14% GovReport perplexity reduction while preserving short-context performance.
Zitieren
APA
Qiao, Y., & Huang, S. (2026). Q-ROAR: Outlier-Aware Rescaling for RoPE Position Interpolation in Quantized Long-Context LLMs (Student Abstract). Proceedings of the AAAI Conference on Artificial Intelligence, 40(48), 41359-41361. https://doi.org/10.1609/aaai.v40i48.42269
Vancouver
Qiao Y, Huang S. Q-ROAR: Outlier-Aware Rescaling for RoPE Position Interpolation in Quantized Long-Context LLMs (Student Abstract). Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(48):41359-41361. doi:10.1609/aaai.v40i48.42269
BibTeX
@article{Qiao2026,
author = {Qiao, Ye and Huang, Sitao},
title = {Q-ROAR: Outlier-Aware Rescaling for RoPE Position Interpolation in Quantized Long-Context LLMs (Student Abstract)},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {48},
pages = {41359--41361},
doi = {10.1609/aaai.v40i48.42269}
}
EUVIMED