DYNAMIC MUSIC RECOMMENDATION THROUGH FACIAL SENTIMENT RECOGNITION AND NEURAL MODELLING

Abstract

Emotion recognition has become essential for creating more natural and empathetic human computer interactions. This project develops a multimodal chatbot system that detects user emotions from text, facial expressions, and voice inputs, and delivers personalized music and story recommendations to improve the user’s mood. The system utilizes the pre-trained RoBERTa model for accurate text-based emotion classification into categories such as joy, sadness, anger, fear, love, and surprise. For voice input, it employs the wav2vec2 model to analyze audio signals and identify emotional states. The chatbot further integrates real-time webcam-based facial emotion detection to support multimodal analysis. Based on the combined or individual emotion predictions, the system intelligently recommends suitable songs and storybooks through external links, helping users enhance their emotional well-being. The entire application is built as a responsive web interface using the Django framework, enabling seamless user interaction through text chat, voice input, and live camera feed. This project demonstrates the practical integration of state-of-the-art transformer models with web technologies to build an intelligent, emotion-aware chatbot. The system provides an effective solution for real-time emotion detection and personalized content recommendation, making human-computer interaction more responsive and supportive in everyday scenarios. Keywords: Emotion Recognition, Multimodal Chatbot, RoBERTa, wav2vec2, Facial Emotion Detection, Music Recommendation System, Human-Computer Interaction.