Isolated Arabic Sign Language Recognition Using a Transformer-based Model and Landmark Keypoints
ACM Transactions on Asian and Low-Resource Language Information Processing · 31.01.2024 · 23 (1) · 1-19
DOI: 10.1145/3584984 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Asian and Low-Resource Language Information Processing, online erschienen am 15.01.2024. Verfasst von Sarah Alyami und 2 weiteren Autor:innen. Die Arbeit wurde bisher 88-mal zitiert (Crossref) und nennt 64 Literaturangaben. Gefördert von: Saudi Data and AI Authority (SDAIA) and King Fahd University of Petroleum and Minerals (KFUPM) under the SDAIA-KFUPM Joint Research Center for Artificial Intel….
Abstract (Originalsprache)
Pose-based approaches for sign language recognition provide light-weight and fast models that can be adopted in real-time applications. This article presents a framework for isolated Arabic sign language recognition using hand and face keypoints. We employed MediaPipe pose estimator for extracting the keypoints of sign gestures in the video stream. Using the extracted keypoints, three models were proposed for sign language recognition: Long-Term Short Memory, Temporal Convolution Networks, and Transformer-based models. Moreover, we investigated the importance of non-manual features for sign language recognition systems and the obtained results showed that combining hand and face keypoints boosted the recognition accuracy by around 4% compared with only hand keypoints. The proposed models were evaluated on Arabic and Argentinian sign languages. Using the KArSL-100 dataset, the proposed pose-based Transformer achieved the highest accuracy of 99.74% and 68.2% in signer-dependent and -independent modes, respectively. Additionally, the Transformer was evaluated on the LSA64 dataset and obtained an accuracy of 98.25% and 91.09% in signer-dependent and -independent modes, respectively. Consequently, the pose-based Transformer outperformed the state-of-the-art techniques on both datasets using keypoints from the signer’s hands and face.
Förderung
- Saudi Data and AI Authority (SDAIA) and King Fahd University of Petroleum and Minerals (KFUPM) under the SDAIA-KFUPM Joint Research Center for Artificial Intel…
Zitieren
APA
Alyami, S., Luqman, H., & Hammoudeh, M. (2024). Isolated Arabic Sign Language Recognition Using a Transformer-based Model and Landmark Keypoints. ACM Transactions on Asian and Low-Resource Language Information Processing, 23(1), 1-19. https://doi.org/10.1145/3584984
Vancouver
Alyami S, Luqman H, Hammoudeh M. Isolated Arabic Sign Language Recognition Using a Transformer-based Model and Landmark Keypoints. ACM Transactions on Asian and Low-Resource Language Information Processing. 2024;23(1):1-19. doi:10.1145/3584984
BibTeX
@article{Alyami2024,
author = {Alyami, Sarah and Luqman, Hamzah and Hammoudeh, Mohammad},
title = {Isolated Arabic Sign Language Recognition Using a Transformer-based Model and Landmark Keypoints},
journal = {ACM Transactions on Asian and Low-Resource Language Information Processing},
year = {2024},
volume = {23},
number = {1},
pages = {1--19},
doi = {10.1145/3584984}
}
EUVIMED