Improving End-to-End Sign Language Translation With Adaptive Video Representation Enhanced Transformer
IEEE Transactions on Circuits and Systems for Video Technology · 2024-09 · 34 (9) · 8327-8342
DOI: 10.1109/tcsvt.2024.3376404 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2024-09. Verfasst von Zidong Liu und 7 weiteren Autor:innen. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 60 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, Innovation Project of Jiangsu Province.
Abstract (Originalsprache)
The aim of end-to-end sign language translation (SLT) is to interpret continuous sign language (SL) video sequences into coherent natural language sentences without any intermediary annotations, i.e., glosses. However, end-to-end SLT suffers several intractable issues: (i) the temporal correspondence constraint loss problem between SL videos and glosses, and (ii) the weakly supervised sequence labeling problem between long SL videos and sentences. To address these issues, we propose an adaptive video representation enhanced Transformer (AVRET), with three extra modules: adaptive masking (AM), local clip self-attention (LCSA) and adaptive fusion (AF). Specifically, we utilize the first AM module to generate a special mask that adaptively drops out temporally important SL video frame representations to enhance the SL video features. Then, we pass the masked video feature to the Transformer encoder consisting of LCSA and masked self-attention to learn clip-level and continuous video-level feature information. Finally, the output feature of encoder is fused with the temporal feature of AM module via the AF module and use the second AM module to generate more robust feature representations. Besides, we add weakly supervised loss terms to constrain these two AM modules. To promote the Chinese SLT research, we further construct CSL-FocusOn, a Chinese continuous SLT dataset, and share its collection method. It involves many common scenarios, and provides SL sentence annotations and multi-cue images of signers. Our experiments on the CSL-FocusOn, PHOENIX14T, and CSL-Daily datasets show that the proposed method achieves the competitive performance on the end-to-end SLT task without using glosses in training. The code is available at https://github.com/LzDddd/AVRET.
Förderung
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- Innovation Project of Jiangsu Province
Zitieren
APA
Liu, Z., Wu, J., Shen, Z., Chen, X., Wu, Q., Gui, Z., Senhadji, L., & Shu, H. (2024). Improving End-to-End Sign Language Translation With Adaptive Video Representation Enhanced Transformer. IEEE Transactions on Circuits and Systems for Video Technology, 34(9), 8327-8342. https://doi.org/10.1109/tcsvt.2024.3376404
Vancouver
Liu Z, Wu J, Shen Z, Chen X, Wu Q, Gui Z, et al. Improving End-to-End Sign Language Translation With Adaptive Video Representation Enhanced Transformer. IEEE Transactions on Circuits and Systems for Video Technology. 2024;34(9):8327-8342. doi:10.1109/tcsvt.2024.3376404
BibTeX
@article{Liu2024,
author = {Liu, Zidong and Wu, Jiasong and Shen, Zeyu and Chen, Xin and Wu, Qianyu and Gui, Zhiguo and Senhadji, Lotfi and Shu, Huazhong},
title = {Improving End-to-End Sign Language Translation With Adaptive Video Representation Enhanced Transformer},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2024},
volume = {34},
number = {9},
pages = {8327--8342},
doi = {10.1109/tcsvt.2024.3376404}
}
EUVIMED