Multi-Stream General and Graph-Based Deep Neural Networks for Skeleton-Based Sign Language Recognition
Electronics · 27.06.2023 · 12 (13) · 2841
DOI: 10.3390/electronics12132841 ↗Kurz gefasst
Fachartikel in der Zeitschrift Electronics, online erschienen am 27.06.2023. Verfasst von Abu Saleh Musa Miah und 4 weiteren Autor:innen. Die Arbeit wurde bisher 36-mal zitiert (Crossref) und nennt 67 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Grand Information Technology Research Center support program, Competitive Research Fund of The University of Aizu, Japan.
Abstract (Originalsprache)
Sign language recognition (SLR) aims to bridge speech-impaired and general communities by recognizing signs from given videos. However, due to the complex background, light illumination, and subject structures in videos, researchers still face challenges in developing effective SLR systems. Many researchers have recently sought to develop skeleton-based sign language recognition systems to overcome the subject and background variation in hand gesture sign videos. However, skeleton-based SLR is still under exploration, mainly due to a lack of information and hand key point annotations. More recently, researchers have included body and face information along with hand gesture information for SLR; however, the obtained performance accuracy and generalizability properties remain unsatisfactory. In this paper, we propose a multi-stream graph-based deep neural network (SL-GDN) for a skeleton-based SLR system in order to overcome the above-mentioned problems. The main purpose of the proposed SL-GDN approach is to improve the generalizability and performance accuracy of the SLR system while maintaining a low computational cost based on the human body pose in the form of 2D landmark locations. We first construct a skeleton graph based on 27 whole-body key points selected among 67 key points to address the high computational cost problem. Then, we utilize the multi-stream SL-GDN to extract features from the whole-body skeleton graph considering four streams. Finally, we concatenate the four different features and apply a classification module to refine the features and recognize corresponding sign classes. Our data-driven graph construction method increases the system’s flexibility and brings high generalizability, allowing it to adapt to varied data. We use two large-scale benchmark SLR data sets to evaluate the proposed model: The Turkish Sign Language data set (AUTSL) and Chinese Sign Language (CSL). The reported performance accuracy results demonstrate the outstanding ability of the proposed model, and we believe that it will be considered a great innovation in the SLR domain.
Förderung
- Grand Information Technology Research Center support program
- Competitive Research Fund of The University of Aizu, Japan
Zitieren
APA
Miah, A. S. M., Hasan, M. A. M., Jang, S., Lee, H., & Shin, J. (2023). Multi-Stream General and Graph-Based Deep Neural Networks for Skeleton-Based Sign Language Recognition. Electronics, 12(13), 2841. https://doi.org/10.3390/electronics12132841
Vancouver
Miah ASM, Hasan MAM, Jang S, Lee H, Shin J. Multi-Stream General and Graph-Based Deep Neural Networks for Skeleton-Based Sign Language Recognition. Electronics. 2023;12(13):2841. doi:10.3390/electronics12132841
BibTeX
@article{Miah2023,
author = {Miah, Abu Saleh Musa and Hasan, Md. Al Mehedi and Jang, Si-Woong and Lee, Hyoun-Sup and Shin, Jungpil},
title = {Multi-Stream General and Graph-Based Deep Neural Networks for Skeleton-Based Sign Language Recognition},
journal = {Electronics},
year = {2023},
volume = {12},
number = {13},
pages = {2841},
doi = {10.3390/electronics12132841}
}
EUVIMED