EUVIMEDEUVIMED entdecken
Crossref

Self-Supervised Representation Learning With Spatial-Temporal Consistency for Sign Language Recognition

Weichao Zhao, Wengang Zhou, Hezhen Hu, Min Wang, Houqiang Li

IEEE Transactions on Image Processing · 2024 · 33 · 4188-4201

DOI: 10.1109/tip.2024.3416881 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2024. Verfasst von Weichao Zhao und 4 weiteren Autor:innen. Die Arbeit wurde bisher 23-mal zitiert (Crossref) und nennt 76 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Youth Innovation Promotion Association Chinese Academy of Sciences, Graphics Processing Unit (GPU) cluster built by the Multimedia Computing and Communication (MCC) Laboratory of Information Science and Technology Institution,….

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recently, there have been efforts to improve the performance in sign language recognition by designing self-supervised learning methods. However, these methods capture limited information from sign pose data in a frame-wise learning manner, leading to sub-optimal solutions. To this end, we propose a simple yet effective self-supervised contrastive learning framework to excavate rich context via spatial-temporal consistency from two distinct perspectives and learn instance discriminative representation for sign language recognition. On one hand, since the semantics of sign language are expressed by the cooperation of fine-grained hands and coarse-grained trunks, we utilize both granularity information and encode them into latent spaces. The consistency between hand and trunk features is constrained to encourage learning consistent representation of instance samples. On the other hand, inspired by the complementary property of motion and joint modalities, we first introduce first-order motion information into sign language modeling. Additionally, we further bridge the interaction between the embedding spaces of both modalities, facilitating bidirectional knowledge transfer to enhance sign language representation. Our method is evaluated with extensive experiments on four public benchmarks, and achieves new state-of-the-art performance with a notable margin. The source code is publicly available at https://github.com/sakura/Code.

Förderung

  • National Natural Science Foundation of China
  • Youth Innovation Promotion Association Chinese Academy of Sciences
  • Graphics Processing Unit (GPU) cluster built by the Multimedia Computing and Communication (MCC) Laboratory of Information Science and Technology Institution,…
  • Supercomputing Center of the USTC

Zitieren

APA

Zhao, W., Zhou, W., Hu, H., Wang, M., & Li, H. (2024). Self-Supervised Representation Learning With Spatial-Temporal Consistency for Sign Language Recognition. IEEE Transactions on Image Processing, 33, 4188-4201. https://doi.org/10.1109/tip.2024.3416881

Vancouver

Zhao W, Zhou W, Hu H, Wang M, Li H. Self-Supervised Representation Learning With Spatial-Temporal Consistency for Sign Language Recognition. IEEE Transactions on Image Processing. 2024;33:4188-4201. doi:10.1109/tip.2024.3416881

BibTeX
@article{Zhao2024,
  author = {Zhao, Weichao and Zhou, Wengang and Hu, Hezhen and Wang, Min and Li, Houqiang},
  title = {Self-Supervised Representation Learning With Spatial-Temporal Consistency for Sign Language Recognition},
  journal = {IEEE Transactions on Image Processing},
  year = {2024},
  volume = {33},
  pages = {4188--4201},
  doi = {10.1109/tip.2024.3416881}
}
Kostenlos starten →