EUVIMEDEUVIMED entdecken
Crossref

Scaling up Multimodal Pre-Training for Sign Language Understanding

Wengang Zhou, Weichao Zhao, Hezhen Hu, Zecheng Li, Houqiang Li

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-12 · 47 (12) · 11753-11767

DOI: 10.1109/tpami.2025.3599313 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-12. Verfasst von Wengang Zhou und 4 weiteren Autor:innen. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 95 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Youth Innovation Promotion Association of the Chinese Academy of Sciences, GPU.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Sign language pre-training (SLP) has significantly improved the performance of diverse sign language understanding (SLU) tasks. However, many existing methods employ pre-training techniques that are tailored to a specific task with small data scale, resulting in limited model generalization. Some others focus solely on exploring visual cues, neglecting semantically textual cues embedded in sign translation texts. These limitations inherently diminish the representative capacity of pre-trained models. To this end, we present a multimodal SLP framework to leverage rich visual contextual information and vision-language semantic consistency with massively available data to enhance the representative capability of sign language video. Specifically, we first curate a large-scale text-labeled sign pose dataset ($\sim$∼1.5M), namely SL-1.5M, from various sources to alleviate the scarcity of pre-training data. Subsequently, we propose a pre-training framework, which integrates sign-text contrastive learning with masked pose modeling as the pretext task. In this way, our framework is empowered to effectively capture contextual cues within sign pose sequences and learn visual representation by aligning semantical text-rich features in a latent space. Moreover, in order to grasp the comprehensive meaning of sign language videos, we concurrently model manual and non-manual information to ensure the holistic integrity of visual content. To validate the generalization and superiority of our proposed pre-trained framework, we conduct extensive experiments without intricate design on diverse SLU tasks, achieving new state-of-the-art performance on multiple benchmarks.

Förderung

  • National Natural Science Foundation of China
  • Youth Innovation Promotion Association of the Chinese Academy of Sciences
  • GPU
  • MCC Lab of Information Science and Technology Institution
  • University of Science and Technology of China
  • National Supercomputing Center, Korea Institute of Science and Technology Information

Zitieren

APA

Zhou, W., Zhao, W., Hu, H., Li, Z., & Li, H. (2025). Scaling up Multimodal Pre-Training for Sign Language Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(12), 11753-11767. https://doi.org/10.1109/tpami.2025.3599313

Vancouver

Zhou W, Zhao W, Hu H, Li Z, Li H. Scaling up Multimodal Pre-Training for Sign Language Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(12):11753-11767. doi:10.1109/tpami.2025.3599313

BibTeX
@article{Zhou2025,
  author = {Zhou, Wengang and Zhao, Weichao and Hu, Hezhen and Li, Zecheng and Li, Houqiang},
  title = {Scaling up Multimodal Pre-Training for Sign Language Understanding},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {12},
  pages = {11753--11767},
  doi = {10.1109/tpami.2025.3599313}
}
Kostenlos starten →