Intelligent Audio‑To‑Indian Sign‑Language and Sign‑To‑Audio Translation System Using MediaPipe, TensorFlow, Whisper and Google TTS
Mansoura Engineering Journal · 01.01.2026 · 51 (4)
DOI: 10.58491/2735-4202.3465 ↗Kurz gefasst
Fachartikel in der Zeitschrift Mansoura Engineering Journal, online erschienen am 01.01.2026. Verfasst von N. Pothanna und 4 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Communication barriers between the Deaf community and the general public remain a persistent challenge, particularly in multilingual settings where Indian Sign Language (ISL) is the primary mode of interaction. We propose a real-time bidirectional translation framework integrating MediaPipe for hand landmark extraction, a Convolutional Neural Network (CNN) classifier built with TensorFlow/Keras for gesture recognition, OpenAI Whisper for robust multilingual speech-to-text transcription, and Google TTS for naturalistic speech synthesis. The system achieves 93% classification accuracy across 35 ISL classes (alphabets A–Z and numerals 1–9) on a dataset of 42,700 images, with macro and weighted F1-scores of 0.93. Deployed as a Flask web application, it supports sign-to-text, text-to-speech, and text-to-sign translation within a single interface, enabling seamless two-way communication between hearing and non-hearing users without requiring specialist hardware
Zitieren
APA
Pothanna, N., Kusuma, T., Haindavi, S., Sreeja, T., & Taj, S. S. (2026). Intelligent Audio‑To‑Indian Sign‑Language and Sign‑To‑Audio Translation System Using MediaPipe, TensorFlow, Whisper and Google TTS. Mansoura Engineering Journal, 51(4). https://doi.org/10.58491/2735-4202.3465
Vancouver
Pothanna N, Kusuma T, Haindavi S, Sreeja T, Taj SS. Intelligent Audio‑To‑Indian Sign‑Language and Sign‑To‑Audio Translation System Using MediaPipe, TensorFlow, Whisper and Google TTS. Mansoura Engineering Journal. 2026;51(4). doi:10.58491/2735-4202.3465
BibTeX
@article{Pothanna2026,
author = {Pothanna, N. and Kusuma, T. and Haindavi, S. and Sreeja, T. and Taj, Shaik Shaheem},
title = {Intelligent Audio‑To‑Indian Sign‑Language and Sign‑To‑Audio Translation System Using MediaPipe, TensorFlow, Whisper and Google TTS},
journal = {Mansoura Engineering Journal},
year = {2026},
volume = {51},
number = {4},
doi = {10.58491/2735-4202.3465}
}
EUVIMED