Prompted Contrastive Learning for Skeleton-Based Action Recognition
IEEE Transactions on Circuits and Systems for Video Technology · 2026-08 · 36 (8) · 11141-11154
DOI: 10.1109/tcsvt.2026.3680904 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2026-08. Verfasst von Taiyi Su und 2 weiteren Autor:innen. Die Arbeit nennt 62 Literaturangaben. Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
Large pre-trained vision-language models have shown great potential across various visual understanding tasks. However, in skeleton-based action recognition, it is challenging to design suitable prompts to guide the model in understanding actions depicted by skeletal structures. Moreover, the modality gap between skeleton data and text descriptions poses obstacles to effective cross-modality contrastive learning. In this work, we propose a prompted contrastive learning framework for skeleton-based action recognition. Specifically, to automatically generate natural language descriptions of actions, we introduce prompted contrast with knowledge engine (PCKE), which utilizes a pre-trained large language model as the knowledge engine to generate text prompts for the input of text encoder. Moreover, as the prompt words generated by the language model may not always be optimal, we explore the potential of the model to autonomously learn prompts. To this end, we present prompted contrast with learning engine (PCLE), a straightforward method that replaces the prompts’ context words with learnable vectors. After separately encoding skeleton data and text prompts with the skeleton and text encoder, a skeleton-language interaction process is implemented to optimize feature alignment and reduce modality gap. The interaction module contains two components: a skeleton-language bridger to connect the representation spaces of the skeleton and text encoders, and cross-modality attention to fuse information of both modalities, facilitating cross-modal knowledge transfer and refining feature alignment. Extensive experiments on four benchmark datasets of NTU RGB+D, NTU RGB+D 120, NW-UCLA, and PKU-MMD demonstrate the effectiveness of the proposed method. The source code of this work can be found in https://mic.tongji.edu.cn
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Su, T., Liu, Z., & Wang, H. (2026). Prompted Contrastive Learning for Skeleton-Based Action Recognition. IEEE Transactions on Circuits and Systems for Video Technology, 36(8), 11141-11154. https://doi.org/10.1109/tcsvt.2026.3680904
Vancouver
Su T, Liu Z, Wang H. Prompted Contrastive Learning for Skeleton-Based Action Recognition. IEEE Transactions on Circuits and Systems for Video Technology. 2026;36(8):11141-11154. doi:10.1109/tcsvt.2026.3680904
BibTeX
@article{Su2026,
author = {Su, Taiyi and Liu, Zheng and Wang, Hanli},
title = {Prompted Contrastive Learning for Skeleton-Based Action Recognition},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2026},
volume = {36},
number = {8},
pages = {11141--11154},
doi = {10.1109/tcsvt.2026.3680904}
}
EUVIMED