Towards Long-window Anchoring in Vision-Language Model Distillation
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (34) · 28955-28963
DOI: 10.1609/aaai.v40i34.40131 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Haoyi Zhou und 5 weiteren Autor:innen.
Abstract (Originalsprache)
While large vision-language models (VLMs) demonstrate impressive long-context understanding, their prevalent small branches fails on linguistics-photography alignment for limited window size. We discover that knowledge distillation improve students capability as compelementary to Rotary Position Embeddings (RoPE) on certain windows size (anchored from large models). Building on this insight, we propose LAid, which explicitly targets the transfer of long-range attention mechanisms through two complementary components: (1) a progressive distance-weighted attention matching that dynamically emphasizes longer position differences during training, and (2) a learnable RoPE response gain modulation that selectively amplifies position sensitivity where needed. Extensive experiments across multiple model families demonstrate that LAid-distilled models achieve up to 3.2× longer effective context windows compared to baseline small models, while maintaining or improving performance on standard VL benchmarks. Spectral analysis also suggests that LAid successfully preserves crucial low-frequency attention components that conventional methods fail to transfer. Our work not only provides practical techniques for building more efficient long-context VLMs but also offers theoretical insights into how positional understanding emerges and transfers during distillation.
Zitieren
APA
Zhou, H., Li, S., Chen, T., Song, Q., Gao, C., & Li, J. (2026). Towards Long-window Anchoring in Vision-Language Model Distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(34), 28955-28963. https://doi.org/10.1609/aaai.v40i34.40131
Vancouver
Zhou H, Li S, Chen T, Song Q, Gao C, Li J. Towards Long-window Anchoring in Vision-Language Model Distillation. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(34):28955-28963. doi:10.1609/aaai.v40i34.40131
BibTeX
@article{Zhou2026,
author = {Zhou, Haoyi and Li, Shuo and Chen, Tianyu and Song, Qi and Gao, Chonghan and Li, Jianxin},
title = {Towards Long-window Anchoring in Vision-Language Model Distillation},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {34},
pages = {28955--28963},
doi = {10.1609/aaai.v40i34.40131}
}
EUVIMED