EUVIMEDEUVIMED entdecken
Crossref

KanAVNet: A CNN-BiLSTM-CTC-Based Audio-Visual Speech Recognition System for Kannada to Assist the Hearing Impaired

Divya, Suresha D.

International Journal of Image, Graphics and Signal Processing · 08.04.2026 · 18 (2) · 87-106

DOI: 10.5815/ijigsp.2026.02.06 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Image, Graphics and Signal Processing, erschienen am 08.04.2026. Verfasst von Divya und 1 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This research outlines a comprehensive dual-modality speech recognition system designed specifically to support hearing-impaired students in understanding spoken Kannada through synchronized processing of auditory signals and visual articulatory cues. The approach capitalizes on deep learning capabilities to improve performance to extract speech-related features from spectrograms and Mel-Frequency Cepstral Coefficients (MFCC) for audio, and lip movement discriminative features via CNNs and Temporal Convolutional Networks (TCNs) for visual input. A hybrid architecture, KanAVNet (Kannada Audio-Visual Network), based on a CNN–BiLSTM framework is integrated with a Connectionist Temporal Classification (CTC) loss function to enable robust sequence-to-sequence mapping while addressing temporal alignment challenges in audio-visual speech recognition. The system is fitted on a custom-developed Kannada audiovisual dataset, addressing the scarcity of regional-language AVSR resources. Empirical evidence shows that the model performs with a high degree of accuracy of 93.2%, a Word Error Rate (WER) of 9.8%, and an F1 score of 91.2%, outperforming baseline unimodal and existing multimodal models. This research highlights the effectiveness of multimodal fusion strategies in noisy environments and showcases the potential of AI-driven tools in promoting accessible and inclusive education for students with auditory impairments.

Zitieren

APA

Divya, & D., S. (2026). KanAVNet: A CNN-BiLSTM-CTC-Based Audio-Visual Speech Recognition System for Kannada to Assist the Hearing Impaired. International Journal of Image, Graphics and Signal Processing, 18(2), 87-106. https://doi.org/10.5815/ijigsp.2026.02.06

Vancouver

Divya, D. S. KanAVNet: A CNN-BiLSTM-CTC-Based Audio-Visual Speech Recognition System for Kannada to Assist the Hearing Impaired. International Journal of Image, Graphics and Signal Processing. 2026;18(2):87-106. doi:10.5815/ijigsp.2026.02.06

BibTeX
@article{Divya2026,
  author = {Divya and D., Suresha},
  title = {KanAVNet: A CNN-BiLSTM-CTC-Based Audio-Visual Speech Recognition System for Kannada to Assist the Hearing Impaired},
  journal = {International Journal of Image, Graphics and Signal Processing},
  year = {2026},
  volume = {18},
  number = {2},
  pages = {87--106},
  doi = {10.5815/ijigsp.2026.02.06}
}
Kostenlos starten →