Overview

The goal of this project was intended to serve as a proof of concept for a machine learning model capable of receiving real time video input and displaying the digit or letter associated with that gesture from American Sign Language (ASL). This could be used in a variety of settings where waiting for an interpreter would not be feasible or degrade services rendered. One example would be the emergency room in a hospital.

Dataset

A dataset was originally procured from Kaggle. I was originally hopeful about this dataset, as it contained clean images of the hands in all 10 positions. The images were used to train a relatively simple CNN model, and initially, when tested against a subset of unseen images, it performed quite well. This would have satisfied the project requirement; however, when this model was applied to a live stream of images from the laptop's webcam, it failed to identify them reliably. The difference between the training set and what was being used during deployment was too large. To address this, a new dataset was generated using the laptop’s camera to ensure the training data matched the deployment data.

Kaggle American Sign Language Dataset

Custom Generated Dataset

Preprocessing

This new dataset required preprocessing that was already applied to the Kaggle dataset from earlier. A preprocessing pipeline was developed to remove excess information and standardize the images being fed to the model. The image process went as follows:

Model

Due to computational requirements and time constraints the model was limited to a classification between 0 and 1 as a proof of concept. A CNN model with a batch size of 32 and 2 classes was implemented. 90% of data was used to train with 5% for validation and 5% for testing. The model contained the following layers:

The model was kept relatively simple to ensure computational time did not exceed a reasonable amount and iteration was possible. As this was a class project that was going far beyond what was required from the project itself the time spent ensuring the model could reasonably identify the test subset of the data while also generalizing to the real time deployment was limited. The above model was capable of achieving a majority success rate on the test dataset while also performing well in real time.

Outcome

The outcome of this project was a model that successfully took a live video stream and identified 0 and 1. While the model would stick to its last prediction when the current frame represents neither of the categories being evaluated, it is robust in its ability to produce the correct value for the input. Due to time constraints, the scope was scaled back, as this working model already met the requirements of the class project.

For this project I developed all technical components. while another member created the presentation and a third created the following report which provides an in depth breakdown of the project and its results.

View Project Report