Are you ready to bridge communication gaps using technology? In this article, we will guide you through setting up a Sign Language Interpreter using Deep Learning, developed as part of HackUNT-19. This innovative project harnesses live video feed from a camera to interpret American Sign Language (ASL) and aims to enhance accessibility for the 70 million deaf individuals worldwide.

Table of Contents

• General Info
• Screenshots
• Demo
• Technologies and Tools
• Setup
• Process
• Code Examples
• Features
• Status
• Contact

General Info

The theme at HackUNT-19 centered on using technology to improve accessibility. Our project seeks to provide a personal translator for deaf individuals, significantly enhancing their communication independence.

Demo

Want to see it in action? Check out the entire demo of the project on YouTube.

Screenshots

Technologies and Tools

• Python
• TensorFlow
• Keras
• OpenCV

Setup

To set up your environment, follow these steps:

• Use the command prompt to execute the installation of required packages:

python -m pip install -r install_packages.txt

Process

Let’s explore how to set up and run the interpreter:

• Run set_hand_histogram.py to create a hand histogram for gesture detection.
• Save the histogram in the code folder or use our pre-created histogram here.
• Add gestures using create_gestures.py, and store them in a database.
• Create variations of the captured gestures by using Rotate_images.py.
• Run load_images.py to split data into training, validation, and test sets.
• Use display_gestures.py to view all captured gestures.
• Train the model through cnn_model_train.py.
• Finally, execute final.py to open the gesture recognition window using your webcam.

Code Examples

Let’s break down the model training using CNN into a tasty analogy. Imagine you are baking a cake. You gather your ingredients (data points) like flour (input shape), sugar (activation functions), and eggs (layers) and blend them together to form the batter (model). The oven represents the training phase and the right temperature signifies fine-tuning our parameters, making sure the cake rises perfectly, just like our model becoming accurately trained.

import numpy as np
from keras import optimizers, Sequential
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D
from keras.utils import np_utils
from keras.callbacks import ModelCheckpoint
from keras import backend as K

def cnn_model():
    model = Sequential()
    model.add(Conv2D(16, (2,2), input_shape=(image_x, image_y, 1), activation='relu'))
    # additional layers here...
    model.compile(loss='categorical_crossentropy', optimizer=optimizers.SGD(lr=1e-2), metrics=['accuracy'])
    return model

Features

Our model predicts 44 ASL characters with an impressive accuracy of 95%. Future enhancements could include:

• Deploying the project on the cloud and creating an API.
• Expanding the vocabulary of signs.
• Incorporating a feedback mechanism for robustness.
• Supporting more sign languages.

Status

This project is finished and we proudly emerged as the winners of the UNT Hackathon 2019. Explore our submission post on devpost.

Contact

Created by me and my teammates:

• Siddharth Oza
• Ashish Sharma
• Manish Shukla

Troubleshooting

If you encounter issues, verify the following:

• Ensure all required packages are correctly installed.
• Check if your webcam is functioning and accessible.
• Review the paths for any input/image files to ensure they’re accurate.
• Consult the community for support or share your challenges.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.