EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Enhancing Kazakh Sign Language Recognition with BiLSTM Using YOLO Keypoints and Optical Flow

Zholdas Buribayev, Maria Aouani, Zhansaya Zhangabay, Ainur Yerkos, Zemfira Abdirazak, Mukhtar Zhassuzak

Applied Sciences · 20.05.2025 · 15 (10) · 5685

DOI: 10.3390/app15105685 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 20.05.2025. Verfasst von Zholdas Buribayev und 5 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Committee of Science of the Ministry of Science and Higher Education of the Republic of Kazakhstan.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Sign languages are characterized by complex and subtle hand movements, which are challenging for computer vision systems to accurately recognize. This study suggests an innovative deep learning pipeline specifically designed for reliable gesture recognition of Kazakh Sign Language. This approach combines key point detection using the YOLO model, optical flow estimation, and a bidirectional long short-term memory (BiLSTM) network. At the initial stage, a dataset is generated using MediaPipe, which is then used to train the YOLO model in order to accurately identify key hand points. After training, the YOLO model extracts key points and bounding boxes from video recordings of gestures, creating consistent representations of movements. To improve the recognition of dynamic gestures, the optical flow is calculated in an area covering 10% of the area around key points, which allows the dynamics of movements to be captured and provides additional time characteristics. The BiLSTM network is trained on multimodal input that combines data on keypoints, bounding boxes, and optical flow, resulting in improved gesture classification accuracy. The experimental results demonstrate that the proposed approach is superior to traditional methods based solely on key points, especially in recognizing fast and complex gestures. The proposed structure promotes the development of sign language recognition technologies, especially for poorly studied languages such as Kazakh, paving the way to more inclusive and effective communication tools.

Förderung

  • Committee of Science of the Ministry of Science and Higher Education of the Republic of Kazakhstan

Zitieren

APA

Buribayev, Z., Aouani, M., Zhangabay, Z., Yerkos, A., Abdirazak, Z., & Zhassuzak, M. (2025). Enhancing Kazakh Sign Language Recognition with BiLSTM Using YOLO Keypoints and Optical Flow. Applied Sciences, 15(10), 5685. https://doi.org/10.3390/app15105685

Vancouver

Buribayev Z, Aouani M, Zhangabay Z, Yerkos A, Abdirazak Z, Zhassuzak M. Enhancing Kazakh Sign Language Recognition with BiLSTM Using YOLO Keypoints and Optical Flow. Applied Sciences. 2025;15(10):5685. doi:10.3390/app15105685

BibTeX
@article{Buribayev2025,
  author = {Buribayev, Zholdas and Aouani, Maria and Zhangabay, Zhansaya and Yerkos, Ainur and Abdirazak, Zemfira and Zhassuzak, Mukhtar},
  title = {Enhancing Kazakh Sign Language Recognition with BiLSTM Using YOLO Keypoints and Optical Flow},
  journal = {Applied Sciences},
  year = {2025},
  volume = {15},
  number = {10},
  pages = {5685},
  doi = {10.3390/app15105685}
}
Kostenlos starten →