EUVIMEDEUVIMED entdecken
Crossref

Q-ROAR: Outlier-Aware Rescaling for RoPE Position Interpolation in Quantized Long-Context LLMs (Student Abstract)

Ye Qiao, Sitao Huang

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (48) · 41359-41361

DOI: 10.1609/aaai.v40i48.42269 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Ye Qiao und 1 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Extending LLM context windows is key for long-range tasks. RoPE-based position interpolation (PI) scales input length without retraining, and post-training quantization (PTQ) enables efficient deployment; however, combining PI with PTQ degrades accuracy due to long-context aliasing, dynamic-range dilation, axis-grid anisotropy, and outlier shifts that induce position-dependent logit noise. We give the first systematic analysis of PI+PTQ and propose two diagnostics: Interpolation Pressure (per-band phase-scaling sensitivity) and Tail Inflation Ratio (outlier shift from short to long contexts). We then introduce Q-ROAR, a RoPE-aware, weight-only stabilization that bands RoPE dimensions and lightly searches per-band scales for W_Q,W_K, with an optional symmetric variant. Q-ROAR needs only a tiny long-context dev set and no fine-tuning or kernel changes, recovering up to 0.7% accuracy and more than 14% GovReport perplexity reduction while preserving short-context performance.

Zitieren

APA

Qiao, Y., & Huang, S. (2026). Q-ROAR: Outlier-Aware Rescaling for RoPE Position Interpolation in Quantized Long-Context LLMs (Student Abstract). Proceedings of the AAAI Conference on Artificial Intelligence, 40(48), 41359-41361. https://doi.org/10.1609/aaai.v40i48.42269

Vancouver

Qiao Y, Huang S. Q-ROAR: Outlier-Aware Rescaling for RoPE Position Interpolation in Quantized Long-Context LLMs (Student Abstract). Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(48):41359-41361. doi:10.1609/aaai.v40i48.42269

BibTeX
@article{Qiao2026,
  author = {Qiao, Ye and Huang, Sitao},
  title = {Q-ROAR: Outlier-Aware Rescaling for RoPE Position Interpolation in Quantized Long-Context LLMs (Student Abstract)},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {48},
  pages = {41359--41361},
  doi = {10.1609/aaai.v40i48.42269}
}
Kostenlos starten →