EUVIMEDEUVIMED entdecken
Crossref

Learning to Prompt With Refining Text Knowledge for Zero-Shot Video Action Recognition

Hao Wang, Fang Liu, Licheng Jiao, Jiahao Wang, Shuo Li, Lingling Li, Puhua Chen, Xu Liu

IEEE Transactions on Multimedia · 2026 · 28 · 4638-4651

DOI: 10.1109/tmm.2026.3660143 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Multimedia, 2026. Verfasst von Hao Wang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref). Gefördert von: Key Project of National Natural Science Foundation of China, Joint Fund Project of National Natural Science Foundation of China, Postdoctoral Fellowship Program of China Postdoctoral Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Foundational vision-language models (VLMs) like CLIP are redefining the vision domain with their exceptional generalization capabilities. Prompt-based learning methods adapt pre-trained VLMs to video action recognition tasks using task-specific learnable text tokens. However, these tokens often struggle to generalize to unseen categories, as they tend to forget general textual knowledge. To address this, we construct knowledge prompts composed of handcrafted and descriptive prompts and introduce a novel knowledge-guided context mapping to enhance the generalization of learnable prompts to unseen categories. This approach mitigates the forgetting of fundamental knowledge by reducing the discrepancy between learnable prompts and knowledge prompts while simultaneously allowing the prompts to extract rich contextual knowledge from LLM data. Then, incorporating the knowledge-guided context mapping into the contrastive loss enables zero-shot transfer of prompts to new categories and data, providing discriminative prompts for both seen and unseen tasks. In addition, we propose an advanced temporal aggregation method that refines uniform mean pooling by incorporating frame-level textual relevance scoring. Extensive evaluations on multiple benchmarks demonstrate that learning to prompt with refining text knowledge is an effective quick-tuning method, achieving superior sample generalization performance without increasing training parameters.

Förderung

  • Key Project of National Natural Science Foundation of China
  • Joint Fund Project of National Natural Science Foundation of China
  • Postdoctoral Fellowship Program of China Postdoctoral Science Foundation
  • Fund for Foreign Scholars in University Research and Teaching Programs
  • Program for Cheung Kong Scholars and Innovative Research Team in University
  • Key Scientific Technological Innovation Research Project by Ministry of Education
  • National Key Laboratory of Human-Machine Hybrid Augmented Intelligence
  • Xi’an Jiaotong University

Zitieren

APA

Wang, H., Liu, F., Jiao, L., Wang, J., Li, S., Li, L., Chen, P., & Liu, X. (2026). Learning to Prompt With Refining Text Knowledge for Zero-Shot Video Action Recognition. IEEE Transactions on Multimedia, 28, 4638-4651. https://doi.org/10.1109/tmm.2026.3660143

Vancouver

Wang H, Liu F, Jiao L, Wang J, Li S, Li L, et al. Learning to Prompt With Refining Text Knowledge for Zero-Shot Video Action Recognition. IEEE Transactions on Multimedia. 2026;28:4638-4651. doi:10.1109/tmm.2026.3660143

BibTeX
@article{Wang2026,
  author = {Wang, Hao and Liu, Fang and Jiao, Licheng and Wang, Jiahao and Li, Shuo and Li, Lingling and Chen, Puhua and Liu, Xu},
  title = {Learning to Prompt With Refining Text Knowledge for Zero-Shot Video Action Recognition},
  journal = {IEEE Transactions on Multimedia},
  year = {2026},
  volume = {28},
  pages = {4638--4651},
  doi = {10.1109/tmm.2026.3660143}
}
Kostenlos starten →