KanAVNet: A CNN-BiLSTM-CTC-Based Audio-Visual Speech Recognition System for Kannada to Assist the Hearing Impaired
International Journal of Image, Graphics and Signal Processing · 08.04.2026 · 18 (2) · 87-106
DOI: 10.5815/ijigsp.2026.02.06 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Image, Graphics and Signal Processing, erschienen am 08.04.2026. Verfasst von Divya und 1 weiteren Autor:innen.
Abstract (Originalsprache)
This research outlines a comprehensive dual-modality speech recognition system designed specifically to support hearing-impaired students in understanding spoken Kannada through synchronized processing of auditory signals and visual articulatory cues. The approach capitalizes on deep learning capabilities to improve performance to extract speech-related features from spectrograms and Mel-Frequency Cepstral Coefficients (MFCC) for audio, and lip movement discriminative features via CNNs and Temporal Convolutional Networks (TCNs) for visual input. A hybrid architecture, KanAVNet (Kannada Audio-Visual Network), based on a CNN–BiLSTM framework is integrated with a Connectionist Temporal Classification (CTC) loss function to enable robust sequence-to-sequence mapping while addressing temporal alignment challenges in audio-visual speech recognition. The system is fitted on a custom-developed Kannada audiovisual dataset, addressing the scarcity of regional-language AVSR resources. Empirical evidence shows that the model performs with a high degree of accuracy of 93.2%, a Word Error Rate (WER) of 9.8%, and an F1 score of 91.2%, outperforming baseline unimodal and existing multimodal models. This research highlights the effectiveness of multimodal fusion strategies in noisy environments and showcases the potential of AI-driven tools in promoting accessible and inclusive education for students with auditory impairments.
Zitieren
APA
Divya, & D., S. (2026). KanAVNet: A CNN-BiLSTM-CTC-Based Audio-Visual Speech Recognition System for Kannada to Assist the Hearing Impaired. International Journal of Image, Graphics and Signal Processing, 18(2), 87-106. https://doi.org/10.5815/ijigsp.2026.02.06
Vancouver
Divya, D. S. KanAVNet: A CNN-BiLSTM-CTC-Based Audio-Visual Speech Recognition System for Kannada to Assist the Hearing Impaired. International Journal of Image, Graphics and Signal Processing. 2026;18(2):87-106. doi:10.5815/ijigsp.2026.02.06
BibTeX
@article{Divya2026,
author = {Divya and D., Suresha},
title = {KanAVNet: A CNN-BiLSTM-CTC-Based Audio-Visual Speech Recognition System for Kannada to Assist the Hearing Impaired},
journal = {International Journal of Image, Graphics and Signal Processing},
year = {2026},
volume = {18},
number = {2},
pages = {87--106},
doi = {10.5815/ijigsp.2026.02.06}
}
EUVIMED