EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Bangla Sign Language Recognition With Multimodal Deep Learning Fusion

Adib Hasan, Mahmodul Hasan Jobayer, Md Abdullah Al Mahmud Pias, Tahmidul Alam, Riasat Khan

Engineering Reports · 2025-04 · 7 (4)

DOI: 10.1002/eng2.70139 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Engineering Reports, online erschienen am 27.04.2025. Verfasst von Adib Hasan und 4 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Sign languages are conveyed through hand gestures and body language; different cultures and languages have unique sign language representations. It is difficult for the general population to interpret all these variations. A Bangla sign language recognition system can help mitigate this problem. In Bangladesh, approximately 13.7 million people are affected by hearing impairments, highlighting the importance of enhancing the Bangla sign language recognition system. In this study, a merged dataset consists of 200‐word classes and 7,000 videos performed by 16 signers. A unique aspect of this dataset is that it includes two modalities. Initially, we extracted frames and audio from the videos. We then employed two frameworks to capture body keypoints from the frames. Additionally, we contributed to expanding the dataset by creating a custom dataset that adds more samples per class. During preprocessing, we augmented the training data, improving our model's learning capacity and reducing overfitting. We developed a system leveraging various deep learning techniques, simultaneously working with two different modalities. We proposed a custom CNN‐LSTM model for pose estimation and integrated VGGish and 2D CNN techniques to process audio in our multimodal model. The CNN‐LSTM model attained an accuracy of 87.08% for pose estimation using the OpenPose framework. The applied ViT model achieved the best performance with 88.52% and 88.39% F1 score to predict body keypoints directly. The VGGish technique delivered the best result for the audio dataset, achieving 81.91% accuracy and 80.42% F1 score. Finally, the late fusion approach combining the ViT and VGGish networks employing the multimodal data achieved the best performance, achieving 94.71% accuracy and 94.52% F1 score.

Zitieren

APA

Hasan, A., Jobayer, M. H., Pias, M. A. A. M., Alam, T., & Khan, R. (2025). Bangla Sign Language Recognition With Multimodal Deep Learning Fusion. Engineering Reports, 7(4). https://doi.org/10.1002/eng2.70139

Vancouver

Hasan A, Jobayer MH, Pias MAAM, Alam T, Khan R. Bangla Sign Language Recognition With Multimodal Deep Learning Fusion. Engineering Reports. 2025;7(4). doi:10.1002/eng2.70139

BibTeX
@article{Hasan2025,
  author = {Hasan, Adib and Jobayer, Mahmodul Hasan and Pias, Md Abdullah Al Mahmud and Alam, Tahmidul and Khan, Riasat},
  title = {Bangla Sign Language Recognition With Multimodal Deep Learning Fusion},
  journal = {Engineering Reports},
  year = {2025},
  volume = {7},
  number = {4},
  doi = {10.1002/eng2.70139}
}
Kostenlos starten →