Intelligent Audio‑To‑Indian Sign‑Language and Sign‑To‑Audio Translation System Using MediaPipe, TensorFlow, Whisper and Google TTS

Authors

• N. Pothanna, Department of Mathematics, Vallurupalli Nageswara Rao Vignana Jyothi Institute of Engineering and Technology, Telangana State, India
• T. Kusuma, Department of Mathematics, Vallurupalli Nageswara Rao Vignana Jyothi Institute of Engineering and Technology, Telangana State, IndiaFollow
• S. Haindavi, Department of CSE (AIML), Vallurupalli Nageswara Rao Vignana Jyothi Institute of Engineering and Technology, Telangana State, India
• T. Sreeja, Department of CSE (AIML), Vallurupalli Nageswara Rao Vignana Jyothi Institute of Engineering and Technology, Telangana State, India.
• Shaik Shaheem Taj, Department of CSE (AIML), Vallurupalli Nageswara Rao Vignana Jyothi Institute of Engineering and Technology, Telangana State, India

Subject Area

Computer Science

Article Type

Special Issue Original Study

Abstract

Communication barriers between the Deaf community and the general public remain a persistent challenge, particularly in multilingual settings where Indian Sign Language (ISL) is the primary mode of interaction. We propose a real-time bidirectional translation framework integrating MediaPipe for hand landmark extraction, a Convolutional Neural Network (CNN) classifier built with TensorFlow/Keras for gesture recognition, OpenAI Whisper for robust multilingual speech-to-text transcription, and Google TTS for naturalistic speech synthesis. The system achieves 93% classification accuracy across 35 ISL classes (alphabets A–Z and numerals 1–9) on a dataset of 42,700 images, with macro and weighted F1-scores of 0.93. Deployed as a Flask web application, it supports sign-to-text, text-to-speech, and text-to-sign translation within a single interface, enabling seamless two-way communication between hearing and non-hearing users without requiring specialist hardware

Keywords

Indian Sign Language, MediaPipe, Tensorflow, Whisper, Google TTS, CNN.

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 License.

Recommended Citation

Pothanna, N.; Kusuma, T.; Haindavi, S.; Sreeja, T.; and Taj, Shaik Shaheem (2026) "Intelligent Audio‑To‑Indian Sign‑Language and Sign‑To‑Audio Translation System Using MediaPipe, TensorFlow, Whisper and Google TTS," Mansoura Engineering Journal: Vol. 51 : Iss. 4 , Article 13.
Available at: https://doi.org/10.58491/2735-4202.3465