Learning to Prompt With Refining Text Knowledge for Zero-Shot Video Action Recognition
IEEE Transactions on Multimedia · 2026 · 28 · 4638-4651
DOI: 10.1109/tmm.2026.3660143 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Multimedia, 2026. Verfasst von Hao Wang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref). Gefördert von: Key Project of National Natural Science Foundation of China, Joint Fund Project of National Natural Science Foundation of China, Postdoctoral Fellowship Program of China Postdoctoral Science Foundation.
Abstract (Originalsprache)
Foundational vision-language models (VLMs) like CLIP are redefining the vision domain with their exceptional generalization capabilities. Prompt-based learning methods adapt pre-trained VLMs to video action recognition tasks using task-specific learnable text tokens. However, these tokens often struggle to generalize to unseen categories, as they tend to forget general textual knowledge. To address this, we construct knowledge prompts composed of handcrafted and descriptive prompts and introduce a novel knowledge-guided context mapping to enhance the generalization of learnable prompts to unseen categories. This approach mitigates the forgetting of fundamental knowledge by reducing the discrepancy between learnable prompts and knowledge prompts while simultaneously allowing the prompts to extract rich contextual knowledge from LLM data. Then, incorporating the knowledge-guided context mapping into the contrastive loss enables zero-shot transfer of prompts to new categories and data, providing discriminative prompts for both seen and unseen tasks. In addition, we propose an advanced temporal aggregation method that refines uniform mean pooling by incorporating frame-level textual relevance scoring. Extensive evaluations on multiple benchmarks demonstrate that learning to prompt with refining text knowledge is an effective quick-tuning method, achieving superior sample generalization performance without increasing training parameters.
Förderung
- Key Project of National Natural Science Foundation of China
- Joint Fund Project of National Natural Science Foundation of China
- Postdoctoral Fellowship Program of China Postdoctoral Science Foundation
- Fund for Foreign Scholars in University Research and Teaching Programs
- Program for Cheung Kong Scholars and Innovative Research Team in University
- Key Scientific Technological Innovation Research Project by Ministry of Education
- National Key Laboratory of Human-Machine Hybrid Augmented Intelligence
- Xi’an Jiaotong University
Zitieren
APA
Wang, H., Liu, F., Jiao, L., Wang, J., Li, S., Li, L., Chen, P., & Liu, X. (2026). Learning to Prompt With Refining Text Knowledge for Zero-Shot Video Action Recognition. IEEE Transactions on Multimedia, 28, 4638-4651. https://doi.org/10.1109/tmm.2026.3660143
Vancouver
Wang H, Liu F, Jiao L, Wang J, Li S, Li L, et al. Learning to Prompt With Refining Text Knowledge for Zero-Shot Video Action Recognition. IEEE Transactions on Multimedia. 2026;28:4638-4651. doi:10.1109/tmm.2026.3660143
BibTeX
@article{Wang2026,
author = {Wang, Hao and Liu, Fang and Jiao, Licheng and Wang, Jiahao and Li, Shuo and Li, Lingling and Chen, Puhua and Liu, Xu},
title = {Learning to Prompt With Refining Text Knowledge for Zero-Shot Video Action Recognition},
journal = {IEEE Transactions on Multimedia},
year = {2026},
volume = {28},
pages = {4638--4651},
doi = {10.1109/tmm.2026.3660143}
}
EUVIMED