BEST: BERT Pre-training for Sign Language Recognition with Coupling Tokenization
Proceedings of the AAAI Conference on Artificial Intelligence · 26.06.2023 · 37 (3) · 3597-3605
DOI: 10.1609/aaai.v37i3.25470 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 26.06.2023. Verfasst von Weichao Zhao und 4 weiteren Autor:innen. Die Arbeit wurde bisher 54-mal zitiert (Crossref).
Abstract (Originalsprache)
In this work, we are dedicated to leveraging the BERT pre-training success and modeling the domain-specific statistics to fertilize the sign language recognition~(SLR) model. Considering the dominance of hand and body in sign language expression, we organize them as pose triplet units and feed them into the Transformer backbone in a frame-wise manner. Pre-training is performed via reconstructing the masked triplet unit from the corrupted input sequence, which learns the hierarchical correlation context cues among internal and external triplet units. Notably, different from the highly semantic word token in BERT, the pose unit is a low-level signal originally locating in continuous space, which prevents the direct adoption of the BERT cross entropy objective. To this end, we bridge this semantic gap via coupling tokenization of the triplet unit. It adaptively extracts the discrete pseudo label from the pose triplet unit, which represents the semantic gesture / body state. After pre-training, we fine-tune the pre-trained encoder on the downstream SLR task, jointly with the newly added task-specific layer. Extensive experiments are conducted to validate the effectiveness of our proposed method, achieving new state-of-the-art performance on all four benchmarks with a notable gain.
Zitieren
APA
Zhao, W., Hu, H., Zhou, W., Shi, J., & Li, H. (2023). BEST: BERT Pre-training for Sign Language Recognition with Coupling Tokenization. Proceedings of the AAAI Conference on Artificial Intelligence, 37(3), 3597-3605. https://doi.org/10.1609/aaai.v37i3.25470
Vancouver
Zhao W, Hu H, Zhou W, Shi J, Li H. BEST: BERT Pre-training for Sign Language Recognition with Coupling Tokenization. Proceedings of the AAAI Conference on Artificial Intelligence. 2023;37(3):3597-3605. doi:10.1609/aaai.v37i3.25470
BibTeX
@article{Zhao2023,
author = {Zhao, Weichao and Hu, Hezhen and Zhou, Wengang and Shi, Jiaxin and Li, Houqiang},
title = {BEST: BERT Pre-training for Sign Language Recognition with Coupling Tokenization},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2023},
volume = {37},
number = {3},
pages = {3597--3605},
doi = {10.1609/aaai.v37i3.25470}
}
EUVIMED