A Hybrid Deep Learning Framework for Real-Time Sign Language Recognition

Abstract

Sign language provides an essential means of communication for people with hearing and speech impairments, but most non-signers cannot interpret hand gestures directly. This work presents a real-time sign language recognition and assistive communication framework that combines computer vision, deep learning, sentence formation, multilingual translation, text-to-speech conversion, and history management. Live hand gestures are captured through a webcam and preprocessed using region-of-interest extraction, resizing, normalization, and background-aware filtering. EfficientNetB0 is used as the primary lightweight feature-extraction and classification backbone for 37 gesture classes comprising the English alphabet, digits, and a blank-space gesture. Recognized characters are assembled into words and sentences, translated into Telugu, Hindi, Kannada, Tamil, or Malayalam, and stored with timestamps in an SQLite history database. The dataset contains 55,500 RGB gesture images, including 44,400 training images and 11,100 validation images. Comparative experiments with Basic CNN, CNN+GRU, and EfficientNetB0 show that EfficientNetB0 provides the strongest model comparison results, recording 96.99% training accuracy, 99.50% validation accuracy, 0.104 training loss, and 0.060 validation loss. The complete framework is deployed through a Streamlit interface for real-time gesture prediction, multilingual output, and communication-log review. The system offers a lightweight and user-friendly platform for educational, healthcare, workplace, and public-service environments. Keywords— Sign Language Recognition, EfficientNetB0, Deep Learning, Computer Vision, Sentence Formation, Multilingual Translation, Streamlit.