EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Speech-to-Sign Gesture Translation for Kazakh: Dataset and Sign Gesture Translation System

Akdaulet Mnuarbek, Akbayan Bekarystankyzy, Mussa Turdalyuly, Dina Oralbekova, Alibek Dyussemkhanov

Computers · 15.03.2026 · 15 (3) · 188

DOI: 10.3390/computers15030188 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computers, online erschienen am 15.03.2026. Verfasst von Akdaulet Mnuarbek und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Science Committee of the Ministry of Science and Higher Education of the Republic of Kazakhstan.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper presents the first prototype of a speech-to-sign language translation system for Kazakh Sign Language (KRSL). The proposed pipeline integrates the NVIDIA FastConformer model for automatic speech recognition (ASR) in the Kazakh language and addresses the challenges of sign language translation in a low-resource setting. Unlike American or British Sign Languages, KRSL lacks publicly available datasets and established translation systems. The pipeline follows a multi-stage process: speech input is converted into text via ASR, segmented into phrases, matched with corresponding gestures, and visualized as sign language. System performance is evaluated using word error rate (WER) for ASR and accuracy metrics for speech-to-sign translation. This study also introduces the first KRSL dataset, consisting of 1200 manually recreated signs, including 95% static images and 5% dynamic gesture videos. To improve robustness under resource-constrained conditions, a Weighted Hybrid Similarity Score (WHSS)-based gesture matching method is proposed. Experimental results show that the FastConformer model achieves an average WER of 10.55%, with 7.8% for isolated words and 13.3% for full sentences. At the phrase level, the system achieves 92.1% accuracy for unigrams, 84.6% for bigrams, and 78.3% for trigrams. The complete pipeline reaches 85% accuracy for individual words and 70% for sentences, with an average latency of 310 ms. These results demonstrate the feasibility and effectiveness of the proposed system for supporting people with hearing and speech impairments in Kazakhstan.

Förderung

  • Science Committee of the Ministry of Science and Higher Education of the Republic of Kazakhstan

Zitieren

APA

Mnuarbek, A., Bekarystankyzy, A., Turdalyuly, M., Oralbekova, D., & Dyussemkhanov, A. (2026). Speech-to-Sign Gesture Translation for Kazakh: Dataset and Sign Gesture Translation System. Computers, 15(3), 188. https://doi.org/10.3390/computers15030188

Vancouver

Mnuarbek A, Bekarystankyzy A, Turdalyuly M, Oralbekova D, Dyussemkhanov A. Speech-to-Sign Gesture Translation for Kazakh: Dataset and Sign Gesture Translation System. Computers. 2026;15(3):188. doi:10.3390/computers15030188

BibTeX
@article{Mnuarbek2026,
  author = {Mnuarbek, Akdaulet and Bekarystankyzy, Akbayan and Turdalyuly, Mussa and Oralbekova, Dina and Dyussemkhanov, Alibek},
  title = {Speech-to-Sign Gesture Translation for Kazakh: Dataset and Sign Gesture Translation System},
  journal = {Computers},
  year = {2026},
  volume = {15},
  number = {3},
  pages = {188},
  doi = {10.3390/computers15030188}
}
Kostenlos starten →