SignFormer-GCN: Continuous sign language translation using spatio-temporal graph convolutional networks
PLOS ONE · 2025
DOI: 10.1371/journal.pone.0316298 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS ONE, 2025. Verfasst von Safaeid Hossain Arib und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Robotics and Mechatronics Engineering, University of Dhaka, Dhaka, Bangladesh. Hauptthemen laut MEDLINE: Sign Language, Neural Networks, Computer, Translating. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 75 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Science and Technology, Government of the People’s Republic of Bangladesh, Conference Travel and Research Grant NSU.
Abstract (Originalsprache)
Sign language is a complex visual language system that uses hand gestures, facial expressions, and body movements to convey meaning. It is the primary means of communication for millions of deaf and hard-of-hearing individuals worldwide. Tracking physical actions, such as hand movements and arm orientation, alongside expressive actions, including facial expressions, mouth movements, eye movements, eyebrow gestures, head movements, and body postures, using only RGB features can be limiting due to discrepancies in backgrounds and signers across different datasets. Despite this limitation, most Sign Language Translation (SLT) research relies solely on RGB features. We used keypoint features, and RGB features to capture better the pose and configuration of body parts involved in sign language actions and complement the RGB features. Similarly, most works on SLT research have used transformers, which are good at capturing broader, high-level context and focusing on the most relevant video frames. Still, the inherent graph structure associated with sign language is neglected and fails to capture low-level details. To solve this, we used a joint encoding technique using a transformer and STGCN architecture to capture the context of sign language expressions and spatial and temporal dependencies on skeleton graphs. Our method, SignFormer-GCN, achieves competitive performance in RWTH-PHOENIX-2014T, How2Sign, and BornilDB v1.0 datasets experimentally, showcasing its effectiveness in enhancing translation accuracy through different sign languages. The code is available at the following link: https://github.com/rabeya-akter/SignLanguageTranslation .
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Robotics and Mechatronics Engineering, University of Dhaka, Dhaka, Bangladesh.
- Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh.
ORCID-Profile: Sejuti Rahman, Shafin Rahman
Förderung
- Ministry of Science and Technology, Government of the People’s Republic of Bangladesh
- Conference Travel and Research Grant NSU
Zitieren
APA
Arib, S. H., Akter, R., Rahman, S., & Rahman, S. (2025). SignFormer-GCN: Continuous sign language translation using spatio-temporal graph convolutional networks. PLOS ONE. https://doi.org/10.1371/journal.pone.0316298
Vancouver
Arib SH, Akter R, Rahman S, Rahman S. SignFormer-GCN: Continuous sign language translation using spatio-temporal graph convolutional networks. PLOS ONE. 2025. doi:10.1371/journal.pone.0316298
BibTeX
@article{Arib2025,
author = {Arib, Safaeid Hossain and Akter, Rabeya and Rahman, Sejuti and Rahman, Shafin},
title = {SignFormer-GCN: Continuous sign language translation using spatio-temporal graph convolutional networks},
journal = {PLOS ONE},
year = {2025},
doi = {10.1371/journal.pone.0316298}
}
EUVIMED