Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs
Mathematics · 30.08.2023 · 11 (17) · 3729
DOI: 10.3390/math11173729 ↗Kurz gefasst
Fachartikel in der Zeitschrift Mathematics, online erschienen am 30.08.2023. Verfasst von Ahmed Mateen Buttar und 5 weiteren Autor:innen. Die Arbeit wurde bisher 108-mal zitiert (Crossref) und nennt 24 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: King Salman Center for Disability Research.
Abstract (Originalsprache)
A speech impairment limits a person’s capacity for oral and auditory communication. A great improvement in communication between the deaf and the general public would be represented by a real-time sign language detector. This work proposes a deep learning-based algorithm that can identify words from a person’s gestures and detect them. There have been many studies on this topic, but the development of static and dynamic sign language recognition models is still a challenging area of research. The difficulty is in obtaining an appropriate model that addresses the challenges of continuous signs that are independent of the signer. Different signers’ speeds, durations, and many other factors make it challenging to create a model with high accuracy and continuity. For the accurate and effective recognition of signs, this study uses two different deep learning-based approaches. We create a real-time American Sign Language detector using the skeleton model, which reliably categorizes continuous signs in sign language in most cases using a deep learning approach. In the second deep learning approach, we create a sign language detector for static signs using YOLOv6. This application is very helpful for sign language users and learners to practice sign language in real time. After training both algorithms separately for static and continuous signs, we create a single algorithm using a hybrid approach. The proposed model, consisting of LSTM with MediaPipe holistic landmarks, achieves around 92% accuracy for different continuous signs, and the YOLOv6 model achieves 96% accuracy over different static signs. Throughout this study, we determine which approach is best for sequential movement detection and for the classification of different signs according to sign language and shows remarkable accuracy in real time.
Förderung
- King Salman Center for Disability Research
Zitieren
APA
Buttar, A. M., Ahmad, U., Gumaei, A. H., Assiri, A., Akbar, M. A., & Alkhamees, B. F. (2023). Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs. Mathematics, 11(17), 3729. https://doi.org/10.3390/math11173729
Vancouver
Buttar AM, Ahmad U, Gumaei AH, Assiri A, Akbar MA, Alkhamees BF. Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs. Mathematics. 2023;11(17):3729. doi:10.3390/math11173729
BibTeX
@article{Buttar2023,
author = {Buttar, Ahmed Mateen and Ahmad, Usama and Gumaei, Abdu H. and Assiri, Adel and Akbar, Muhammad Azeem and Alkhamees, Bader Fahad},
title = {Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs},
journal = {Mathematics},
year = {2023},
volume = {11},
number = {17},
pages = {3729},
doi = {10.3390/math11173729}
}
EUVIMED