Emotion-Driven Ambiance and Music Control Using Facial Expression Recognition

Prasad Bhosle, Sujeet More, Aboli Mohite, Purva Mene, Prathamesh Abnave, Susmita Mali — Trinity College of Engineering and Research, Pune, India

Emophony proposes a solution that uses a network of cameras to track a person's emotions and adjust lighting, music, and ambiance in real time to enhance their mood. The system relies on computer vision and machine learning for facial expression recognition. A mood-based music player recommends songs based on the user's current emotional state, improving customer satisfaction. The project also underscores the growing role of music in daily life and the need for better music emotion recognition methods. It employs an artificial bee colony algorithm to optimize neural network structures, delivering higher recognition accuracy and faster processing.

Introduction

Mental health has become a pressing concern in the twenty-first century, driving people to explore various strategies for understanding and managing their emotions. Against this backdrop, we developed an emotion-based music player. Unlike earlier approaches, a Convolutional Neural Network (CNN) automatically identifies relevant features without human supervision, making it ideal for computer vision and image classification tasks. Because convolutional layers contain fewer parameters than the fully connected layers of a traditional neural network, CNNs process images more efficiently. They also use parameter sharing, which further boosts their effectiveness with visual data.

The system begins when a webcam connected to a microprocessor or other capable device captures images of the user's face. These images are then fed into a trained deep learning model. Recognizing emotions from facial images is challenging because human faces share many similar features, leading to high covariance. Even human observers typically identify another person's mood with only 67% to 73% accuracy. Nevertheless, the system is designed to reliably detect facial moods and deliver correct outputs.

Existing System

Music has been woven into human history and daily life, affecting emotions in contexts ranging from retail and exercise to relaxation and medical care. As traditional music storage and retrieval methods no longer serve personalized needs, emotion-based music classification and retrieval have emerged. Labelling music with emotions poses difficulties, but automatic music emotion recognition technology continues to advance. Various machine learning approaches have achieved recognition rates over 60%. Deep learning based Music Emotion Recognition and Classification (MER) is now an active research field within Music Information Retrieval (MIR). The vast digital library makes manual retrieval and classification impractical, which drives the need for automatic systems especially in diverse, multicultural settings.

Emotions in music connect to nationality, age, mood, purpose, and cultural background — all factors that increase the complexity of MER. The relationship between music and human emotions also has therapeutic applications, often described through neurotransmitters and hormones.

Music serves both as a source of enjoyment and as a medical tool for stress relief. High-end music players now feature volume control, modulation, and genre selection, but the goal is to create a platform that plays music based on a person's current mood using facial expression recognition powered by AI and machine learning. The process begins with face detection, followed by dimensionality reduction and extraction of facial features such as eyes, nose, and mouth to identify emotions. Music influences human emotions and cognitive functions, helping enhance mindfulness and mood. Preferences connect to individual characteristics, and digital emotion detection is increasingly used in music and movies. The proposed recommender system identifies user emotions and suggests appropriate songs. It uses the Kaggle Look Acknowledgment dataset for emotion detection and Bollywood Hindi songs for the music library. Facial emotion detection with a Convolutional Neural Network achieves approximately 95.14% accuracy, helping improve users' emotional states through music recommendations in line with their detected feelings.

Images and videos shared on social networks have created an explosion of online data. Analysing this multimedia at an emotional level has grown in importance, as emotions affect decision-making and have applications from marketing to political forecasting. While text-based sentiment analysis has progressed, analyzing emotions in social images presents challenges. A gap exists between visual features and emotional content — finding features that express emotions in images remains a key difficulty. Earlier efforts mostly predicted dominant emotions common among viewers, without accounting for individual, subjective evaluations.

Methodology

The system uses facial expression recognition to generate playlists that match the user's emotional state. Its main steps and components are as follows:

1. Real-Time Capture: The system captures the user's face accurately in real time using a camera, providing the initial input for emotion recognition.
2. Facial Recognition: The captured face is processed by a Convolutional Neural Network trained to analyze facial features. CNNs automatically and adaptively learn spatial hierarchies of features from input images through convolutional layers. This lets them capture complex patterns and representations, making them effective for image classification, object detection, and segmentation.
3. Emotion Detection: Extracted features from the facial image are used to identify emotions. Feature extraction is critical for classification. After preprocessing, high-intensity features — including eyebrows, forehead wrinkles, nose, jawline, and mouth corners — are extracted using the Local Binary Pattern algorithm. This technique compares each pixel to its neighbors and generates a binary number.

4. Music Recommendation: This module recommends songs based on the user's emotions and desired music ambiance. The playlist responds to emotions such as happy, sad, or angry.

Figure 1 of 2 — System Architecture

CNN Algorithm

A Convolutional Neural Network is a deep learning algorithm well suited for image recognition and processing. It consists of multiple layers: convolutional layers, pooling layers, and fully connected layers. Convolutional layers apply filters to the input image to extract features such as edges, textures, and shapes. These outputs pass through pooling layers, which down-sample feature maps, reducing spatial dimensions while preserving essential information. Then the data move through fully connected layers that classify the image.

CNNs draw on neuroscience research. They comprise layers of artificial neurons — called nodes — that calculate weighted sums of inputs and return activation maps. In a CNN, each node carries weight values. When a layer receives data such as an image, it selects certain visual features based on pixel values. Each layer returns activation maps that highlight important features. For example, an initial layer finds picture edges, then passes this coarse definition to the next layer, which detects corners and color groupings. This process continues through subsequent layers until the network makes a prediction.

Figure 2 of 2 — CNN Architecture

Conclusion

Emophony integrates smoothly with multiple music platforms, helping users find songs suited to their current mood without manually searching for tracks. The system offers a curated list of songs based on users' emotions and directs them to gaana.com, where a single click plays the chosen music. Affective computing methods predict and respond to users' changing emotional states. Emophony provides a diverse selection that enriches the listening experience by aligning music with feeling. Additionally, the paper introduces a Music Information Retrieval (MIR) network designed to classify music according to emotional characteristics.