Moodify - Emotion-Based Music Recommendation App

With the rise of personalized music streaming services, there is a growing need for systems that can recommend music based on users' emotional states. Realizing this need, Moodify is being developed by Son Nguyen in 2024 to provide personalized music recommendations based on users' detected emotions.

The Moodify project is an integrated emotion-based music recommendation system that combines frontend, backend, AI/ML models, and data analytics to provide personalized music recommendations based on user emotions. The application analyzes text, speech, or facial expressions and suggests music that aligns with the detected emotions.

Supporting both desktop and mobile platforms, Moodify offers a seamless user experience with real-time emotion detection and music recommendations. The project leverages React for the frontend, Django for the backend, and three advanced, self-trained AI/ML models for emotion detection. Data analytics scripts are used to visualize emotion trends and model performance.

Table of Contents

• 🎵 Overview
• 🌟 Features
• 🛠️ Technologies
• 🖼️ User Interface
  • Landing Page
  • Home Page
  • Text Input
  • Speech Input
  • Facial Expression Input
  • Profile Page
  • Results/Recommendations Page
  • Login Page
  • Registration Page
  • 404 Not Found Page
  • Footer
• 🌐 Live Deployment
• 📂 Complete File Structure
• 🛠️ Getting Started
  • Prerequisites
  • Setup and Train AI/ML Models
  • Set Up the Backend
  • Install and Run the Frontend
• 📋 API Endpoints
  • User Endpoints
  • Emotion Detection Endpoints
  • Admin Interface Endpoints
  • Admin Interface
• 🚀 Backend APIs Documentation
• 🤖 About the AI/ML Models
  • AI/ML Models Overview
  • Training the AI/ML Models
  • Testing the AI/ML Models
  • Pre-Trained Models
• 📊 Analytics Scripts
• 📱 Mobile App Version
• 🔗 Load Balancing
• 🐳 Containerization
• ☸️ Kubernetes
• 🔗 Jenkins
• 🔧 Contributing
• 📝 License
• 📧 Contact

🎵 Overview

Moodify provides personalized music recommendations based on users' emotional states detected through text, speech, and facial expressions. It interacts with a Django-based backend, AI/ML models for emotion detection, and utilizes data analytics for visual insights into emotion trends and model performance.

🌟 Features

• User registration and login functionality.
• Input analysis through text, speech, and facial expressions.
• Real-time music recommendations based on emotion detection.
• Visualization of emotion detection results and user history.
• Data analytics scripts for emotion trends and model performance.
• AI/ML models for text, speech, and facial emotion detection.
• User profile management and customization.
• Mobile app version for seamless user experience.
• Progressive Web App (PWA) features for offline support.
• Admin panel for managing users, recommendations, and data analytics.

🌐 Live Deployment

The Moodify app is currently live and deployed on Vercel. You can access the live app using the following link: Moodify.

Feel free to also visit the backend at Moodify Backend API.

Diclaimer: The backend of Moodify is currently hosted with the Free Tier of Render, so it may take a few seconds to load initially. Additionally, it may spin down after a period of inactivity or high traffic, so please be patient if the backend takes a few seconds to respond.

Also, the amount of memory allocated by Render is only 512MB with 0.1 CPU, so the backend may run out of memory if there are too many requests at once, which may cause the server to restart. Also, the facial and speech emotion detection models may also fail due to memory constraints - which can also cause the server to restart.

There is no guarantee of uptime or performance with the current deployment, unless I have more resources (money) to upgrade the server :( Feel free to contact me if you encounter any issues or need further assistance.

🛠️ Technologies

• Frontend:
  • React
  • Axios
  • Material UI (MUI)
  • React Router
  • React Redux
  • Jest and React Testing Library
• Backend:
  • Django
  • Django REST Framework
  • MongoEngine
  • JWT Authentication
  • Spotify APIs
  • Swagger and Redoc for API documentation
• Databases:
  • MongoDB (with MongoDB Atlas)
  • Redis (for server-side caching)
  • SQLite
• AI/ML Models:
  • PyTorch
  • TensorFlow
  • Keras
  • HuggingFace Transformers
  • Pandas
  • Scikit-learn
  • NumPy
  • FER (Facial Expression Recognition) Library
• Load Balancing:
  • NGINX
  • Gunicorn
• Data Analytics:
  • Pandas
  • Matplotlib
  • Hadoop
  • Spark
• Mobile:
  • React Native
  • Expo
  • Expo Go
• PWA:
  • Progressive Web App features for offline support
• Containerization, Deployment, and CI/CD:
  • Docker
  • Kubernetes
  • Jenkins
  • Render
  • Vercel

🖼️ User Interface

Landing Page

Landing Page - Dark Mode

Home Page

Home Page - Dark Mode

Text Input

Text Input - Dark Mode

Speech Input

Speech Input - Dark Mode

Facial Expression Input

Facial Expression Input - Dark Mode

Profile Page

Profile Page - Dark Mode

Results - Recommendations Page

Results - Recommendations Page - Dark Mode

Login Page

Login Page - Dark Mode

Registration Page

Registration Page - Dark Mode

Forgot Password Page

Forgot Password Page - Dark Mode

Reset Password - Once the User Has Verified Their Account Ownership

Reset Password - Dark Mode

Privacy Policy Page

Privacy Policy Page - Dark Mode

Terms of Service Page

Terms of Service Page - Dark Mode

404 Not Found Page

404 Not Found Page - Dark Mode

Responsive Design - Mobile View

Footer

📂 Complete File Structure

The project has a comprehensive file structure combining frontend, backend, AI/ML models, and data analytics components:

Moodify-Emotion-Music-App/
├── frontend/                      # React frontend for the web application
│   ├── public/
│   │   ├── index.html             # Main HTML file
│   │   ├── manifest.json          # Web app manifest
│   │   └── favicon.ico            # Favicon for the app
│   │
│   ├── src/
│   │   ├── components/            # Contains all React components
│   │   ├── pages/                 # Contains main pages of the app
│   │   ├── styles/                # Contains global styles and themes
│   │   ├── context/               # Contains React Context API
│   │   ├── App.js                 # Main App component
│   │   ├── index.js               # Entry point for React
│   │   └── theme.js               # Material UI theme configuration
│   │ 
│   ├── .gitignore                 # Git ignore file
│   ├── Dockerfile                 # Dockerfile for containerization
│   ├── package.json               # NPM dependencies and scripts
│   └── README.md                  # Project documentation
│ 
├── backend/                       # Django backend for API services and database management
│   ├── manage.py                  # Django's command-line utility
│   ├── requirements.txt           # Backend dependencies
│   ├── backend/
│   │   ├── settings.py            # Django settings for the project
│   │   ├── urls.py                # URL declarations for the project
│   │   ├── users/                 # User management components
│   │   └── api/                   # Emotion detection and recommendation APIs
│   │
│   ├── .gitignore                 # Git ignore file
│   ├── Dockerfile                 # Dockerfile for containerization
│   └── db.sqlite3                 # SQLite database (if used)
│
├── ai_ml/                         # AI/ML models for emotion detection
│   ├── data/                      # Datasets for training and testing
│   ├── models/                    # Trained models for emotion detection
│   ├── src/                       # Source files for emotion detection and recommendation
│   │   ├── api/                   # API scripts for running emotion detection services
│   │   ├── recommendation/        # Music recommendation logic
│   │   └── data_processing/       # Data preprocessing scripts
│   │
│   ├── Dockerfile                 # Dockerfile for containerization
│   └── README.md                  # AI/ML documentation
│
├── data_analytics/                # Data analytics scripts and visualizations
│   ├── emotion_distribution.py    # Script for visualizing emotion distribution
│   ├── training_visualization.py  # Script for visualizing training and validation metrics
│   ├── predictions_analysis.py    # Script for analyzing model predictions
│   ├── recommendation_analysis.py # Script for visualizing music recommendations
│   ├── spark-hadoop/              # Spark and Hadoop integration scripts
│   └── visualizations/            # Generated visualizations
│
├── kubernetes/                    # Kubernetes deployment files
│   ├── backend-deployment.yaml    # Deployment file for the backend service
│   ├── backend-service.yaml       # Deployment file for the backend service
│   ├── frontend-deployment.yaml   # Deployment file for the frontend service
│   ├── frontend-service.yaml      # Deployment file for the frontend service
│   └── configmap.yaml             # ConfigMap for environment variables
│
├── mobile/                        # React Native mobile application
│   ├── App.js                     # Main entry point for React Native app
│   ├── index.js                   # App registry for React Native
│   ├── package.json               # NPM dependencies and scripts
│   ├── components/                # React Native components
│   │   ├── Footer.js              # Footer component
│   │   ├── Navbar.js              # Header component
│   │   ├── Auth/                  # Authentication components (e.g., Login, Register)
│   │   └── Profile/               # Profile-related components
│   │
│   ├── context/                   # React Context API for state management
│   │   └── DarkModeContext.js     # Dark mode context provider
│   │
│   ├── pages/                     # Main pages of the app
│   │   ├── HomePage.js            # Home page component
│   │   ├── ProfilePage.js         # Profile page component
│   │   ├── ResultsPage.js         # Results page component
│   │   ├── LandingPage.js         # Landing page component
│   │   └── (and more...)
│   │
│   ├── assets/                    # Images, fonts, and other assets
│   ├── styles/                    # Styling files (similar to CSS for web)
│   ├── .gitignore                 # Git ignore file
│   ├── package.json               # Dependencies and scripts
│   └── README.md                  # Mobile app documentation
│
├── nginx/                         # NGINX configuration files (for load balancing and reverse proxy)
│   ├── nginx.conf                 # Main NGINX configuration file
│   └── Dockerfile                 # Dockerfile for NGINX container
│
├── images/                        # Images used in the README documentation 
├── docker-compose.yml             # Docker Compose file for containerization
└── README.md                      # Comprehensive README file for the entire project

🛠️ Getting Started

Prerequisites

• Node.js (v14 or higher)
• Python 3.8 or later
• MongoDB
• Virtual Environment (venv)
• .env File (for environment variables - you create your own credentials following the example file or contact me for mine.)

1. Setup and Train AI/ML Models

Start with setting up and training the AI/ML models, as they will be required for the backend to function properly.

Or, you can download the pre-trained models from the Google Drive links provided in the Pre-Trained Models section. If you choose to do so, you can skip this section for now.

1. Clone the repository:

   git clone https://github.com/hoangsonww/Moodify-Emotion-Music-App.git

2. Navigate to the AI/ML directory:

   cd Moodify-Emotion-Music-App/ai_ml

3. Create and activate a virtual environment:

   python -m venv venv
   source venv/bin/activate   # For macOS/Linux
   .\venv\Scripts\activate    # For Windows

4. Install dependencies:

   pip install -r requirements.txt

5. Edit the configurations in the src/config.py file:

   • Visit the src/config.py file and update the configurations as needed, especially your Spotify API keys and configure ALL the paths.
   • Visit the individual model training scripts in the src/models directory and update the paths to the datasets and output paths as needed.
   • Ensure all paths are correctly set before training the models!

6. Train the text emotion model:

   python src/models/train_text_emotion.py

   Repeat similar commands for other models as needed (e.g., facial and speech emotion models).

7. Ensure all trained models are placed in the models directory, and that you have trained all necessary models before moving to the next step!

8. Test the trained AI/ML models as needed:

   • Run the src/models/test_emotion_models.py script to test the trained models.
   • Ensure the models are providing accurate predictions before moving to the next step.

2. Set Up the Backend

Once the AI/ML models are ready, proceed with setting up the backend.

1. Navigate to the backend directory:

   cd ../backend

2. Create and activate a virtual environment:

   python -m venv venv
   source venv/bin/activate   # For macOS/Linux
   .\venv\Scripts\activate    # For Windows

3. Install dependencies:

   pip install -r requirements.txt

4. Configure your secrets and environment:

   • Create a .env file in the backend directory.
   • Add the following environment variables to the .env file:

     SECRET_KEY=your_secret_key
     DEBUG=True
     ALLOWED_HOSTS=<your_hosts>
     MONGODB_URI=<your_mongodb_uri>
     

   • Visit backend/settings.py and add SECRET_KEY & set DEBUG to True.
   • Important: Ensure these steps are completed before running the backend server.

5. Run database migrations:

   python manage.py migrate

6. Start the Django server:

   python manage.py runserver

   The backend server will be running at http://127.0.0.1:8000/.

3. Install and Run the Frontend

Finally, set up the frontend to interact with the backend.

1. Navigate to the frontend directory:

   cd ../frontend

2. Install dependencies using Yarn:

   npm install

3. Start the development server:

   npm start

   The frontend will start at http://localhost:3000.

Note: If you encounter any problems or need my .env file, feel free to contact me.

📋 API Endpoints

User Endpoints

HTTP Method	Endpoint	Description
POST	/users/register/	Register a new user
POST	/users/login/	Login a user and obtain a JWT token
GET	/users/user/profile/	Retrieve the authenticated user's profile
PUT	/users/user/profile/update/	Update the authenticated user's profile
DELETE	/users/user/profile/delete/	Delete the authenticated user's profile
POST	/users/recommendations/	Save recommendations for a user
GET	/users/recommendations/<str:username>/	Retrieve recommendations for a user by username
DELETE	/users/recommendations/<str:username>/<str:recommendation_id>/	Delete a specific recommendation for a user
DELETE	/users/recommendations/<str:username>/	Delete all recommendations for a user
POST	/users/mood_history/<str:user_id>/	Add a mood to the user's mood history
GET	/users/mood_history/<str:user_id>/	Retrieve mood history for a user
DELETE	/users/mood_history/<str:user_id>/	Delete a specific mood from the user's history
POST	/users/listening_history/<str:user_id>/	Add a track to the user's listening history
GET	/users/listening_history/<str:user_id>/	Retrieve listening history for a user
DELETE	/users/listening_history/<str:user_id>/	Delete a specific track from the user's history
POST	/users/user_recommendations/<str:user_id>/	Save a user's recommendations
GET	/users/user_recommendations/<str:user_id>/	Retrieve a user's recommendations
DELETE	/users/user_recommendations/<str:user_id>/	Delete all recommendations for a user
POST	/users/verify-username-email/	Verify if a username and email are valid
POST	/users/reset-password/	Reset a user's password
GET	/users/verify-token/	Verify a user's token

Emotion Detection Endpoints

HTTP Method	Endpoint	Description
POST	/api/text_emotion/	Analyze text for emotional content
POST	/api/speech_emotion/	Analyze speech for emotional content
POST	/api/facial_emotion/	Analyze facial expressions for emotions
POST	/api/music_recommendation/	Get music recommendations based on emotion

Admin Interface Endpoints

HTTP Method	Endpoint	Description
GET	/admin/	Access the Django Admin interface

Documentation Endpoints

HTTP Method	Endpoint	Description
GET	/swagger/	Access the Swagger UI API documentation
GET	/redoc/	Access the Redoc API documentation
GET	/	Access the API root endpoint (Swagger UI)

Admin Interface

1. Create a superuser:

   python manage.py createsuperuser

2. Access the admin panel at http://127.0.0.1:8000/admin/

3. You should see the following login page:

🚀 Backend APIs Documentation

Our backend APIs are all well-documented using Swagger UI and Redoc. You can access the API documentation at the following URLs:

• Swagger UI: https://moodify-emotion-music-app.onrender.com/swagger.
• Redoc: https://moodify-emotion-music-app.onrender.com/redoc.
• Note that the backend server may spin down after a period of inactivity, so it may take a few seconds to load initially.

Alternatively, you can run the backend server locally and access the API documentation at the following endpoints:

• Swagger UI: http://127.0.0.1:8000/swagger.
• Redoc: http://127.0.0.1:8000/redoc.

Regardless of your choice, you should see the following API documentation if everything is running correctly:

Swagger UI:

Redoc:

🤖 About the AI/ML Models

AI/ML Models Overview

The AI/ML models are built using PyTorch, TensorFlow, Keras, and HuggingFace Transformers. These models are trained on various datasets to detect emotions from text, speech, and facial expressions.

The emotion detection models are used to analyze user inputs and provide real-time music recommendations based on the detected emotions. The models are trained on various datasets to capture the nuances of human emotions and provide accurate predictions.

• Text Emotion Detection: Detects emotions from text inputs.
• Speech Emotion Detection: Analyzes emotions from speech inputs.
• Facial Emotion Detection: Detects emotions from facial expressions.

The models are integrated into the backend API services to provide real-time emotion detection and music recommendations for users.

Training the AI/ML Models

The models must be trained first before using them in the backend services. Ensure that the models are trained and placed in the models directory before running the backend server. Refer to the (Getting Started)[#getting-started] section for more details.

Examples of training the text emotion model.

To train the models, you can run the provided scripts in the ai_ml/src/models directory. These scripts are used to preprocess the data, train the models, and save the trained models for later use. These scripts include:

• train_text_emotion.py: Trains the text emotion detection model.
• train_speech_emotion.py: Trains the speech emotion detection model.
• train_facial_emotion.py: Trains the facial emotion detection model.

Ensure that you have the necessary dependencies, datasets, and configurations set up before training the models. Specifically, make sure to visit the config.py file and update the paths to the datasets and output directories to the correct ones on your system.

Note: By default, these scripts will prioritize using your GPU with CUDA (if available) for faster training. However, if that is not available on your machine, the scripts will automatically fall back to using the CPU for training. To ensure that you have the necessary dependencies for GPU training, install PyTorch with CUDA support using the following command:

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

Testing the AI/ML Models

After that, you can run the test_emotion_models.py script to test the trained models and ensure they are providing accurate predictions:

python src/models/test_emotion_models.py

Alternatively, you can run the simple Flask API to test the models via RESTful API endpoints:

python ai_ml/src/api/emotion_api.py

The endpoints are as follows:

• /text_emotion: Detects emotion from text input
• /speech_emotion: Detects emotion from speech audio
• /facial_emotion: Detects emotion from an image
• /music_recommendation: Provides music recommendations based on the detected emotion

Important: For more information about training and using the models, please refer to the AI/ML documentation in the ai_ml directory.

Pre-Trained Models

However, if training the model is too resource-intensive for you, you can use the following Google Drive links to download the pre-trained models:

• Text Emotion Detection Model - model.safetensors. Please download this model.safetensors file and place it into the ai_ml/models/text_emotion_model directory.
• Speech Emotion Detection Model - scaler.pkl. Please download this and place this into the ai_ml/models/speech_emotion_model directory.
• Speech Emotion Detection Model - trained_speech_emotion_model.pkl. Please download this and place this into the ai_ml/models/speech_emotion_model directory.
• Facial Emotion Detection Model - trained_facial_emotion_model.pt. Please download this and place this into the ai_ml/models/facial_emotion_model directory.

These have been pre-trained on the datasets for you and are ready to use in the backend services or for testing purposes once downloaded and correctly placed in the models directory.

Feel free to contect me if you encounter any issues or need further assistance with the AI/ML models.

📊 Analytics Scripts

The data_analytics folder provides data analysis and visualization scripts to gain insights into the emotion detection model's performance.

1. Run All Analytics Scripts:

   python data_analytics/main.py

2. View generated visualizations in the visualizations folder.

3. Here are some example visualizations:

Emotion Distribution Visualization

Training Loss Curve Visualization

📱 Mobile App Version

There is also a mobile version of the Moodify app built using React Native and Expo. You can find the mobile app in the mobile directory.

1. Navigate to the mobile directory:

   cd ../mobile

2. Install dependencies using Yarn:

    yarn install

3. Start the Expo development server:

   yarn start

4. Scan the QR code using the Expo Go app on your mobile device to run the app.

If successful, you should see the following home screen:

Feel free to explore the mobile app and test its functionalities!

🔗 Load Balancing

The project uses NGINX and Gunicorn for load balancing and serving the Django backend. NGINX acts as a reverse proxy server, while Gunicorn serves the Django application.

1. Install NGINX:

   sudo apt-get update
   sudo apt-get install nginx

2. Install Gunicorn:

    pip install gunicorn

3. Configure NGINX:

   • Update the NGINX configuration file (if needed) at /nginx/nginx.conf with your configuration.

4. Start NGINX and Gunicorn:

   • Start NGINX:

     sudo systemctl start nginx

   • Start Gunicorn:

     gunicorn backend.wsgi:application

5. Access the backend at http://<server_ip>:8000/.

Feel free to customize the NGINX configuration and Gunicorn settings as needed for your deployment.

🐳 Containerization

The project can be containerized using Docker for easy deployment and scaling. You can create Docker images for the frontend, backend, and AI/ML models.

1. Build the Docker images:

   docker compose up --build

2. The Docker images will be built for the frontend, backend, and AI/ML models. Verify the images using:

   docker images

If you encounter any errors, try to rebuild your image without using the cache since Docker's cache may cause issues.

docker-compose build --no-cache

☸️ Kubernetes

We also added Kubernetes deployment files for the backend and frontend services. You can deploy the services on a Kubernetes cluster using the provided YAML files.

1. Deploy the backend service:

   kubectl apply -f kubernetes/backend-deployment.yaml

2. Deploy the frontend service:

    kubectl apply -f kubernetes/frontend-deployment.yaml

3. Expose the services:

   kubectl expose deployment moodify-backend --type=LoadBalancer --port=8000
   kubectl expose deployment moodify-frontend --type=LoadBalancer --port=3000

4. Access the services using the LoadBalancer IP:

   • You can access the backend service at http://<backend_loadbalancer_ip>:8000.
   • You can access the frontend service at http://<frontend_loadbalancer_ip>:3000.

Feel free to visit the kubernetes directory for more information about the deployment files and configurations.

🔗 Jenkins

We have also included Jenkins pipeline script for automating the build and deployment process. You can use Jenkins to automate the CI/CD process for the Moodify app.

1. Install Jenkins on your server or local machine.

2. Create a new Jenkins pipeline job:

   • Create a new pipeline job in Jenkins.
   • Configure the pipeline to use the Jenkinsfile in the jenkins directory.

3. Run the Jenkins pipeline:

   • Run the Jenkins pipeline to build and deploy the Moodify app.
   • The pipeline will automate the build, test, and deployment process for the app.

Feel free to explore the Jenkins pipeline script in the Jenkinsfile and customize it as needed for your deployment process.

🔧 Contributing

• Contributions are welcome! Feel free to fork the repository and submit a pull request.

• Note that this project is still under active development, and any contributions are appreciated.

• If you have any suggestions, feature requests, or bug reports, feel free to open an issue here.

📝 License

• This project is licensed under the MIT License. Please see the LICENSE file for details.

📧 Contact

• Feel free to contact me at [email protected] for any questions or feedback.

---

Happy Coding and Vibin'! 🎶

Created with ❤️ by Son Nguyen in 2024.

---

🔝 Back to Top