Self-Adaptive Vision-Language Tracking With Context Prompting
IEEE Transactions on Image Processing · 2025 · 34 · 8046-8058
DOI: 10.1109/tip.2025.3635016 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2025. Verfasst von Jie Zhao und 5 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 72 Literaturangaben. Gefördert von: National Natural Science Foundations of China, Liaoning Provincial Science and Technology Project, China Postdoctoral Science Foundation.
Abstract (Originalsprache)
Due to the substantial gap between vision and language modalities, along with the mismatch problem between fixed language descriptions and dynamic visual information, existing vision-language tracking methods exhibit performance on par with or slightly worse than vision-only tracking. Effectively exploiting the rich semantics of language to enhance tracking robustness remains an open challenge. To address these issues, we propose a self-adaptive vision-language tracking framework that leverages the pre-trained multi-modal CLIP model to obtain well-aligned visual-language representations. A novel context-aware prompting mechanism is introduced to dynamically adapt linguistic cues based on the evolving visual context during tracking. Specifically, our context prompter extracts dynamic visual features from the current search image and integrates them into the text encoding process, enabling self-updating language embeddings. Furthermore, our framework employs a unified one-stream Transformer architecture, supporting joint training for both vision-only and vision-language tracking scenarios. Our method not only bridges the modality gap but also enhances robustness by allowing language features to evolve with visual context. Extensive experiments on four vision-language tracking benchmarks demonstrate that our method effectively leverages the advantages of language to enhance visual tracking. Our large model can obtain 55.0% AUC on $\text {LaSOT}_{\text {EXT}}$ and 69.0% AUC on TNL2K. Additionally, our language-only tracking model achieves performance comparable to that of state-of-the-art vision-only tracking methods on TNL2K. Code is available at https://github.com/zj5559/SAVLT.
Förderung
- National Natural Science Foundations of China
- Liaoning Provincial Science and Technology Project
- China Postdoctoral Science Foundation
- Funding of the Central Universities of China
Zitieren
APA
Zhao, J., Chen, X., Li, S., Bo, C., Wang, D., & Lu, H. (2025). Self-Adaptive Vision-Language Tracking With Context Prompting. IEEE Transactions on Image Processing, 34, 8046-8058. https://doi.org/10.1109/tip.2025.3635016
Vancouver
Zhao J, Chen X, Li S, Bo C, Wang D, Lu H. Self-Adaptive Vision-Language Tracking With Context Prompting. IEEE Transactions on Image Processing. 2025;34:8046-8058. doi:10.1109/tip.2025.3635016
BibTeX
@article{Zhao2025,
author = {Zhao, Jie and Chen, Xin and Li, Shengming and Bo, Chunjuan and Wang, Dong and Lu, Huchuan},
title = {Self-Adaptive Vision-Language Tracking With Context Prompting},
journal = {IEEE Transactions on Image Processing},
year = {2025},
volume = {34},
pages = {8046--8058},
doi = {10.1109/tip.2025.3635016}
}
EUVIMED