CrossVLS: Cross-Modal Vision-Language Prototypes for Self-Supervised Skeleton Action Representation Learning
Electronics · 28.07.2026 · 15 (15) · 3329
DOI: 10.3390/electronics15153329 ↗Kurz gefasst
Fachartikel in der Zeitschrift Electronics, online erschienen am 28.07.2026. Verfasst von Kenan Ye und 2 weiteren Autor:innen. Die Arbeit nennt 64 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China, Shanghai Municipal Science and Technology Major Project.
Abstract (Originalsprache)
Artificial intelligence (AI)-driven positioning and tracking systems combine geometric trajectories with behavior understanding in smart-city, healthcare, and autonomous environments. Skeleton sequences provide compact, privacy-preserving motion geometry, but labeled data are costly, and coordinate-only self-supervision cannot recover object, scene, or interaction cues. Vision-language transfer can supply these cues, but instance-level targets remain sensitive to noisy crops, incomplete descriptions, and ambiguous actions. We propose CrossVLS, which is a cross-modal vision-language-guided framework that transfers semantic knowledge from red, green, and blue (RGB) frames and generated language descriptions to a skeleton encoder during pretraining while retaining skeleton-only inference. CrossVLS replaces noisy instance-level transfer with a shared prototype space: skeleton, RGB, and language features are softly assigned to a common prototype bank through balanced optimal transport, and the resulting assignments define semantic soft targets for contrastive learning. A full-batch progressive training schedule gradually increases cross-modal guidance without splitting the physical batch, preserving the support set used to construct semantic targets. Experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD demonstrate strong performance under linear and semi-supervised evaluation using only the pretrained skeleton encoder at inference. These results show that prototype-mediated vision-language transfer can improve skeleton representations for the behavior-interpretation stage of positioning and tracking pipelines.
Förderung
- National Key R&D Program of China
- National Natural Science Foundation of China
- Shanghai Municipal Science and Technology Major Project
- Shanghai Science and Technology Innovation Action Plan Project
- Natural Science Foundation of Shanghai
- Shanghai Leading Talent Program of Eastern Talent Plan
- Fundamental Research Funds for the Central Universities
Zitieren
APA
Ye, K., Zhao, S., & Liang, S. (2026). CrossVLS: Cross-Modal Vision-Language Prototypes for Self-Supervised Skeleton Action Representation Learning. Electronics, 15(15), 3329. https://doi.org/10.3390/electronics15153329
Vancouver
Ye K, Zhao S, Liang S. CrossVLS: Cross-Modal Vision-Language Prototypes for Self-Supervised Skeleton Action Representation Learning. Electronics. 2026;15(15):3329. doi:10.3390/electronics15153329
BibTeX
@article{Ye2026,
author = {Ye, Kenan and Zhao, Shengjie and Liang, Shuang},
title = {CrossVLS: Cross-Modal Vision-Language Prototypes for Self-Supervised Skeleton Action Representation Learning},
journal = {Electronics},
year = {2026},
volume = {15},
number = {15},
pages = {3329},
doi = {10.3390/electronics15153329}
}
EUVIMED