Real-Time Sign Language Recognition Using Multi-Modal Landmark Extraction with MediaPipe Holistic
Machines and Algorithms · 16.03.2016 · 5 (01) · 22-32
DOI: 10.66108/mna.v5i01.90 ↗Kurz gefasst
Fachartikel in der Zeitschrift Machines and Algorithms, online erschienen am 16.03.2016. Verfasst von Muhammad Anas und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Sign language is a major mode of communication to millions of deaf and hard-of-hearing people, but unfamiliarity with the access of real-time interpretation technologies still restricts effective interaction and access to information. To overcome this, this paper proposes a real-time Sign Language Recognition (SLR) framework that utilizes a Sign Language Recognition system on top of Google MediaPipe Holistic framework that learns detailed 3D face, hand and whole-body landmarks with the help of standard RGB video input only. The system proposed has stable performance with an approximate real-time frame rate and feature vectors of the 1,662 dimensions being processed on consumer-grade hardware and operating in real time. Compared to current technologies, which utilize specialized sensors or only track hand gestures, the technique provides abundant spatial and temporal data of several body parts at once using only a single webcam. The system pipeline consists of data acquisition, landmark localization (468 facial landmarks, 21 key points per hand and 33 landmarks of the body pose per frame), multimodal feature vector creation and performance analysis using visualization features. Initial experiments show that it is not impossible to identify a small set of basic signs (e.g. greetings, like hello and thanks), having a demonstration of concept, not a trained large-vocabulary classifier. The framework offers an excellent base to integrate supervised temporal models in the future, including recurrent neural networks and transformer-based architectures, and includes low-cost, scalable, and inclusive SLR solutions to real-world problems in education, healthcare, and assistive technologies
Zitieren
APA
Anas, M., & Umar, M. A. (2016). Real-Time Sign Language Recognition Using Multi-Modal Landmark Extraction with MediaPipe Holistic. Machines and Algorithms, 5(01), 22-32. https://doi.org/10.66108/mna.v5i01.90
Vancouver
Anas M, Umar MA. Real-Time Sign Language Recognition Using Multi-Modal Landmark Extraction with MediaPipe Holistic. Machines and Algorithms. 2016;5(01):22-32. doi:10.66108/mna.v5i01.90
BibTeX
@article{Anas2016,
author = {Anas, Muhammad and Umar, Muhammad Azeem},
title = {Real-Time Sign Language Recognition Using Multi-Modal Landmark Extraction with MediaPipe Holistic},
journal = {Machines and Algorithms},
year = {2016},
volume = {5},
number = {01},
pages = {22--32},
doi = {10.66108/mna.v5i01.90}
}
EUVIMED