Speech-to-Sign Gesture Translation for Kazakh: Dataset and Sign Gesture Translation System
Computers · 15.03.2026 · 15 (3) · 188
DOI: 10.3390/computers15030188 ↗Kurz gefasst
Fachartikel in der Zeitschrift Computers, online erschienen am 15.03.2026. Verfasst von Akdaulet Mnuarbek und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Science Committee of the Ministry of Science and Higher Education of the Republic of Kazakhstan.
Abstract (Originalsprache)
This paper presents the first prototype of a speech-to-sign language translation system for Kazakh Sign Language (KRSL). The proposed pipeline integrates the NVIDIA FastConformer model for automatic speech recognition (ASR) in the Kazakh language and addresses the challenges of sign language translation in a low-resource setting. Unlike American or British Sign Languages, KRSL lacks publicly available datasets and established translation systems. The pipeline follows a multi-stage process: speech input is converted into text via ASR, segmented into phrases, matched with corresponding gestures, and visualized as sign language. System performance is evaluated using word error rate (WER) for ASR and accuracy metrics for speech-to-sign translation. This study also introduces the first KRSL dataset, consisting of 1200 manually recreated signs, including 95% static images and 5% dynamic gesture videos. To improve robustness under resource-constrained conditions, a Weighted Hybrid Similarity Score (WHSS)-based gesture matching method is proposed. Experimental results show that the FastConformer model achieves an average WER of 10.55%, with 7.8% for isolated words and 13.3% for full sentences. At the phrase level, the system achieves 92.1% accuracy for unigrams, 84.6% for bigrams, and 78.3% for trigrams. The complete pipeline reaches 85% accuracy for individual words and 70% for sentences, with an average latency of 310 ms. These results demonstrate the feasibility and effectiveness of the proposed system for supporting people with hearing and speech impairments in Kazakhstan.
Förderung
- Science Committee of the Ministry of Science and Higher Education of the Republic of Kazakhstan
Zitieren
APA
Mnuarbek, A., Bekarystankyzy, A., Turdalyuly, M., Oralbekova, D., & Dyussemkhanov, A. (2026). Speech-to-Sign Gesture Translation for Kazakh: Dataset and Sign Gesture Translation System. Computers, 15(3), 188. https://doi.org/10.3390/computers15030188
Vancouver
Mnuarbek A, Bekarystankyzy A, Turdalyuly M, Oralbekova D, Dyussemkhanov A. Speech-to-Sign Gesture Translation for Kazakh: Dataset and Sign Gesture Translation System. Computers. 2026;15(3):188. doi:10.3390/computers15030188
BibTeX
@article{Mnuarbek2026,
author = {Mnuarbek, Akdaulet and Bekarystankyzy, Akbayan and Turdalyuly, Mussa and Oralbekova, Dina and Dyussemkhanov, Alibek},
title = {Speech-to-Sign Gesture Translation for Kazakh: Dataset and Sign Gesture Translation System},
journal = {Computers},
year = {2026},
volume = {15},
number = {3},
pages = {188},
doi = {10.3390/computers15030188}
}
EUVIMED