Task-to-Instance Prompt Learning for Vision-Language Models at Test Time
IEEE Transactions on Image Processing · 2025 · 34 · 1908-1920
DOI: 10.1109/tip.2025.3546840 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2025. Verfasst von Zhihe Lu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 73 Literaturangaben. Gefördert von: Ministry of Education, Singapore, Academic Research Fund Tier 2, National Research Foundation, Singapore.
Abstract (Originalsprache)
Prompt learning has been recently introduced into the adaption of pre-trained vision-language models (VLMs) by tuning a set of trainable tokens to replace hand-crafted text templates. Despite the encouraging results achieved, existing methods largely rely on extra annotated data for training. In this paper, we investigate a more realistic scenario, where only the unlabeled test data is available. Existing test-time prompt learning methods often separately learn a prompt for each test sample. However, relying solely on a single sample heavily limits the performance of the learned prompts, as it neglects the task-level knowledge that can be gained from multiple samples. To that end, we propose a novel test-time prompt learning method of VLMs, called Task-to-Instance PromPt LEarning (TIPPLE), which adopts a two-stage training strategy to leverage both task- and instance-level knowledge. Specifically, we reformulate the effective online pseudo-labeling paradigm along with two tailored components: an auxiliary text classification task and a diversity regularization term, to serve the task-oriented prompt learning. After that, the learned task-level prompt is further combined with a tunable residual for each test sample to integrate with instance-level knowledge. We demonstrate the superior performance of TIPPLE on 15 downstream datasets, e.g., the average improvement of 1.87% over the state-of-the-art method, using ViT-B/16 visual backbone. Our code is open-sourced at https://github.com/zhiheLu/TIPPLE.
Förderung
- Ministry of Education, Singapore
- Academic Research Fund Tier 2
- National Research Foundation, Singapore
- Artificial Intelligence Singapore Program
Zitieren
APA
Lu, Z., Bai, J., Li, X., Xiao, Z., & Wang, X. (2025). Task-to-Instance Prompt Learning for Vision-Language Models at Test Time. IEEE Transactions on Image Processing, 34, 1908-1920. https://doi.org/10.1109/tip.2025.3546840
Vancouver
Lu Z, Bai J, Li X, Xiao Z, Wang X. Task-to-Instance Prompt Learning for Vision-Language Models at Test Time. IEEE Transactions on Image Processing. 2025;34:1908-1920. doi:10.1109/tip.2025.3546840
BibTeX
@article{Lu2025,
author = {Lu, Zhihe and Bai, Jiawang and Li, Xin and Xiao, Zeyu and Wang, Xinchao},
title = {Task-to-Instance Prompt Learning for Vision-Language Models at Test Time},
journal = {IEEE Transactions on Image Processing},
year = {2025},
volume = {34},
pages = {1908--1920},
doi = {10.1109/tip.2025.3546840}
}
EUVIMED