AI 2-WAY COMMUNICATION FOR SIGN LANGUAGE

Abstract

Communication between hearing individuals and people with hearing or speech impairments remains a significant social challenge due to differences in communication methods. Although sign language is widely used within the deaf community, most people are unfamiliar with it, creating barriers in education, healthcare, workplaces, and public services. This paper presents an AI-based two-way communication system that enables seamless interaction between deaf and hearing individuals using computer vision, machine learning, and natural language processing techniques. The proposed system employs MediaPipe to detect hand landmarks from live webcam video and extracts three-dimensional key points representing hand gestures. These features are classified using a TensorFlow-based neural network trained on Indian Sign Language (ISL) gestures. The recognized signs are converted into readable text and can be further refined into grammatically meaningful sentences using a Large Language Model through the Groq LLaMA3 API. In the reverse direction, hearing users communicate by typing text messages that are displayed clearly for deaf users through an accessible interface. The web application is implemented using Flask, OpenCV, HTML, CSS, and JavaScript, providing real-time communication without requiring specialized hardware. Experimental evaluation demonstrates reliable gesture recognition, responsive performance, and effective bidirectional communication. The modular architecture also supports future expansion with additional sign vocabulary and multilingual capabilities, making the proposed system a practical and cost-effective assistive technology for inclusive communication. Keywords: Artificial Intelligence, Indian Sign Language, MediaPipe, TensorFlow, Computer Vision, Natural Language Processing, Flask, TwoWay Communication, Assistive Technology.