miguelramosfdz / sign-language

Android application which uses feature extraction algorithms and machine learning (SVM) to recognise and translate static sign language gestures.

Sign Language

The Sign Language app is an Android application which can translate static ASL and BSL signs, such as the fingerspelling alphabet. These translated signs can be displayed to the user whilst allowing for sentences to be constructed. This app is currently a proof of concept to illustrate low-cost, freely available and offline Sign Language recognition using purely visual data.

• https://github.com/Mquinn960/offline-trainer
• https://github.com/Mquinn960/dataset-creator

The current beta version of this app can be tested here:

• https://play.google.com/store/apps/details?id=mquinn.sign_language

(Click the image below to watch the video demo)

For a comprehensive step by step guide on using these applications and some additional info on how they work, please see my new help repo

Getting Started

• Clone the repo onto your local machine by using:

  HTTPS: git clone https://github.com/Mquinn960/sign-language.git

  SSH: git clone [email protected]:Mquinn960/sign-language.git

• Ensure the prerequisites below are installed/satisfied

• If you're using Android Studio, load the project and hit run

Prerequisites

• Load in a trained SVM XML file created using the Offline Trainer
  • https://github.com/Mquinn960/offline-trainer
  • You must place the file (named trained.xml) in the sign-language\app\src\main\res\raw\ directory. This gets loaded when running the app the first time.
  • Note: If you're just looking for a quick start, then you can download a sample trained.xml here
• Ensure you have an Android smartphone connected to your computer, and that debugging is enabled
• It is recommended that you use the Android Studio IDE noted below

Development Environment

• Created using Android Studio

User Guide

For a comprehensive step by step guide on using these applications and some additional info on how they work, please see my new help repo

• Using this app requires the use of a trained.xml file, which contains the Machine Learning information required to make predictions about your Sign Language gestures
• Follow the instructions found in the Offline Trainer repo to create this file from input image training data you create using the Dataset Creator app, or find online - for more info see the Offline Trainer repo
  • https://github.com/Mquinn960/offline-trainer
  • https://github.com/Mquinn960/dataset-creator
• Once the trained.xml file has been added to the app raw resources folder as per the Prerequisites
• Start the app, and point the smartphone camera at a person performing Sign Language alphabet gestures
• Use the onscreen buttons to capture the translated gestures

Exporting the imaging kernel

If you want to alter the Sign Language app and then use the Sign Language app's imaging kernel to train a new model with the Offline Trainer, you must first run the Grade make-jar task.

• Edit the app's Gradle build file sign-language\app\build.gradle
• Comment out the entire com.android.application build step
• Uncomment the com.android.library task
• Perform a Gradle sync
• Run the make-jar Gradle task
• Find the exported imaging kernel sign-language\app\build\outputs\jar\app-release-null.jar and import this into the Offline Trainer's "new" folder, as described in the repo README
  • https://github.com/Mquinn960/offline-trainer
• (Optional) Run any training you want to do in the Offline Trainer, then take the trained SVM XML file and import it into the Sign Language app as descibed in the User Guide above
• Undo steps 2 and 3 and run the main app task again.

Built With

• Java 8 - Java Programming Language
• Android - Android OS
• Gradle - Gradle build system
• OpenCV - The Open Source Computer Vision Library

Contributing

• Feel free to submit issues to this repository but please include usable information if you are looking to have something fixed
• Use Feature Branching where possible
• Submit Pull Requests to @Mquinn960 for review

Authors

• Matthew Quinn

License

This project is licensed under the MIT License - see the LICENSE.md file for details