Sign Language (ASL) Detection System using MobileNet, TensorFlow v2, Python, Flask, OpenCv2, Web Speech API
![Logo](/priyanka-maz/sign-language-detection/raw/main/static\love-hand-l-fav.png)
![Working Gif](/priyanka-maz/sign-language-detection/raw/main/screenshots/8.Live_ASL_detection_TTS.gif)
This gif shows ASL sign detetcion and recognition of the class which is written as interpreted text.
To hear the text-to-speech, navigate to Live_ASL_detection_TTS.mp4
First we create the virtual environment and clone the repository into it. Then, we install all dependencies before running the application.
python3 -m venv /path/to/venv
cd /path/to/venv
git clone https://github.com/priyanka-maz sign-language-detection
source bin/activate
cd sign-language-detection
python3 -m pip install -r requirements.txt
python3 app.py
-
Click on 'Start' on the homepage to begin sign language detection on your video feed
-
Permissions for video feed usage must be set to 'Allow'
-
Write letters in American Sign Language to form words or phrases which show up on the console on the right
-
Click on 'Play' button below the console to convert your sentences from text-to-speech for seamless communication
-
For non-ASL users, you can refer to the English to ASL converter on the homepage
-
Follow 'Tips' under console for further instructions
- This website was developed using Python, Flask, HTML, CSS & Javascript.
- Users who wish to converse in ASL, may use this detection system to convert the interpreted text to speech for communicating without bounds.
- Users who aren't equipped with ASL may use the English to ASL converter to learn relevant symbols.
- The voice for TTS technology may be changed in settings.
- Developers may check the raw footage opened using SocketIO and the prediction on top of that OpenCV feed.
- Text-to-speech translation uses
SpeechSynthesisUtterance()
of the Web Speech API. TensorFlow 2.0
andKeras
library is used in the development of our model. The final model used for testing and deployment isMobileNet
.- Other machine learning techniques employed are:
- Data Augmentation
- Transfer Learning
- Validation Set for Accuracy metric
- Early Stopping
- Nadam Optimizer
- Exponential Decay Learning Rate
- Categorical Cross Entropy Loss
.
├── app.py.............................Runs the Flask application and deploys all webpages
│ as well as calls necessary computational functions
├── datasets...........................Contains tar.gz file of the dataset and
│ some necessary statistics for the dataset
├── LICENSE............................MIT LICENSE
├── models.............................TFLITE version models of MobileNet and EfficientNet
│ ├── model_efficientnet_v2s.tflite
│ └── model_mobilenet_v2.tflite
├── README.md
├── requirements.txt
├── screenshots........................Contains necessary visualizations about the
│ model performance, data and website
├── static.............................Contains all js, css files and images used in the website
│ ├── connection.js
│ ├── opencv.js
│ └── style.css
├── templates..........................Contains all HTML templates deployed in the website
│ ├── about.html
│ ├── index.html.....................HTML template of the page where sign detection occurs
│ ├── landing.html
│ └── layout.html
├── train.py...........................Used to train the entire model, contains all
│ ML techniques mentioned above
├── webcam_detect.py...................Loads the model and predicts the class from the
│ frame input given as softmax probability which is sent
│ back to the client's end
└── webcam.py..........................Used for testing ASL Sign Detection and preiction locally