anasmorahhib / 3d-cnn-gesture-recognition

Gesture recognition using tensorflow from a large video database

3D-CNN-Gesture-recognition

Gesture recognition via 3D CNN neural network using Tensorflow.
Our project consists in developing a Python language solution, using deep learning techniques, for hand gestures recognition. the idea of this project is to detect the gestures in a three-dimensional space, that is to say that, instead of analyzing the shape of the hand in each image separately, we will analyze the movement of the hand in time, which means analyzing several images at once.
A little example to understand what we want to do, If you raise two fingers, the machine must know it, but the 3D gives us more than that, it will be able to detect if we move the two fingers left or right or other, which makes us analysis the gesture and the movement of the hand too.

The technologies used:

Python 3.6
TensorFlow 2.0 Beta
OpenCV 3

Description to read before you begin:

• 20BN-JESTER: It contains all training, validation and testing data. You have to download it from The 20BN-jester Dataset V1

• training-sample.py: If you use your personal computer, often you will not have the performance needed to work with all this data. this script will help you bring out a sample that you can work with. if you want to work with all data, ignore this file.

• validation-sample.py: the same thing as training-sample.py for validation.

• main-beginners-syntax.py: the main code with a simple syntax.

• main-experts-syntax.py: the main code with a more complex syntax.

• test-app.py: a small application to test the model after training.

Noticeable:

as you know, a deep learning project goes through several steps before starting the training of the model, like the data cleaning, the standardization, the creation of the model ... for this in both files main-beginners-syntax.py and main-experts-syntax.py it is better to use a block-by-block execution using spider for example, or use the file main.ipynb with jupyter notebook.
the advantage of this is that if you got an error or you want to change in a step, you do not need to run all the code each time, just execute the part you want.
I invite you to install Anaconda, to have a good environment.

Demo

Youtube link - https://youtu.be/dMAAyceGKAg

3d-cnn-gesture-recognition's Issues

closed

closed

Regarding training-sample.py

Hi, I understand that training-sample.py is used to extract the desired hand gestures from 20bn jester dataset so that our pc can work lesser. I have downloaded 20bn jester to my pc and tried on your code. However, it does not work well, it shows "Directory not copied, Error: [WinError 3] The system cannot find the path specified: '20bn-jester-v148433' " and so on. Please assist me on this issue.

Ps: after I downloaded all files of 20bn jester dataset, I only extract one file out of 22 files and try the coding. Should I extract all files only I can apply the coding?

Thank you.

Pretrained model

Hi,

It's a really cool project. Thank you for providing the code.

I was wondering if you would be able to provide the pre-trained model.

Please let me know.

Thank you

ValueError: 'Turning Hand Clockwise' is not in list

I face the error below, please help me to check what is the problem, thank you very much!!!

---

ValueError Traceback (most recent call last)
in
13 if len(new_frame) == 15: # partition each training on two trainings.
14 new_frames.append(new_frame) # append each partition to training data
---> 15 training_targets.append(targets_name.index(targets[int(directory)]))
16 counter_training +=1
17 new_frame = []

ValueError: 'Turning Hand Clockwise' is not in list

Custom dataset help

Hello,

I tried creating my own custom dataset (my own 2 second videos and then had frames extracted in folders named with custom video id). Used ffmpeg to extract 12 frames per second. Used 10 videos per class, and 5 for validation (around 1000 images)
I then adapted csv files (train and validation ones) to add the required video ids for each of my custom classes - I tried this with 3 classes.

First issue is that code uses deprecated functions regarding Series.from_csv and failed. I had to convert csv to json files in order to continue.
Then I trained with 20 epochs, then tried test-app.py (after adapting my own class names) but nothing is predicted, no gesture although training results seems ok.

Would it be possible to have like a small guide in here on how to use your example with custom data (file/folder structure/sizes etc) and maybe revisit code so it works with latest libraries etc?

Thank you very much

error coours on handiling data

---

KeyError Traceback (most recent call last)
in
13 if len(new_frame) == 15:
14 new_frames_cv.append(new_frame)
---> 15 cv_targets.append(targets_name.index(targets_validation[int(directory)]))
16 counter_validation +=1
17 new_frame = []

KeyError: 70259

Extracting the dataset

Hi,Can you please help with detailed instructions on how to extract the dataset o windows. The information i am finding online is not helpful.I am a beginner.Your help will be sincerely appreciated.

NaN Loss during training

Hello,

I have an error when training the model: value of train loss was always NaN, value of accuracy wasnot increased. I reused your code to train on google colab but changed target_names. I don't know why it is.

Thank you for your help.

Best regards,

Error in adjusting training data

I am trying to follow along in the Jupyter notebook.

I have ran the training and validation sample scripts, and now in the Jupyter Notebook I get the following error in the "Adjust Training Data" cell

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-48-ffcb8753cb2e> in <module>
     13             if len(new_frame) == 15: # partition each training on two trainings.
     14                 new_frames.append(new_frame) # append each partition to training data
---> 15                 training_targets.append(targets_name.index(targets[int(directory)]))
     16                 counter_training +=1
     17                 new_frame = []

ValueError: 'Turning Hand Clockwise' is not in list

Any ideas?

Kernel dies

Normalisation: training

print('old mean', training_data.mean())
scaler = StandardScaler()
scaled_images = scaler.fit_transform(training_data.reshape(-1, 156464))
print('new mean', scaled_images.mean())
scaled_images = scaled_images.reshape(-1, 15, 64, 64, 1)
print(scaled_images.shape)

When executing this command my kernel dies. Can you give any solution?

Help

Brother , I don't have well configured Laptop so I used this dataset and I have to use only 4 kind of class from data set to control powerpoint. So how can I only get those 4 kind class from whole dataset ?
Actually I want to make a sort dataset from this .

Incorrect gesture recognition

Hi,

I trained the model with a validation subset and training subset using the provided scripts.

The training samples are ~ 1GB and the validation ~ 150 MB.

I trained the model with 10 epochs and the accuracy output was >98%.

However, when running test-app.py the results are not correct. I.e none of my gestures are predicted correctly.

Is there something I may have omitted?

P.S thanks for the code, it's v clean compared to other 3DD-CNN implementations.

Why "partition each training on two trainings"

Could you please tell me why are you "# partition each training on two trainings." when you're adjusting the training and validation data ?

Extraction of frames from the video.

Hello Anas,
I am doing similar project like this but in that project i want to recognize the continous signs of sign language. I found your project helpful and i am just a begginer in Deep-learning. I want to know that i have data of sign language in form of simple videos of gesture performed by volunteers which is similar to 20BN-jester data-set.
I want to know how you are extracting the frames from the videos, and one more thing in my case if there is a video of a whole sentance that include different gestures of signs( below link is to similar sentance video, you can have good idea what i want to say) then how we can extract the frames for the all these signs and identify them as individual sign.
I will be very thankfull for your help.

https://www.psl.org.pk/signs/en/sentences/5c9634bb0625be0004d9217b

Issue in section 7 of Main code!

It gives an error that pandas.Series does not exist and converting it to pd.read_csv() doesn't exactly convert the csv in the dictionary format you wished for!

Error while running train_step()

I encounter following error while trying to train the model with experts main function.

---

InvalidArgumentError: ```
Traceback (most recent call last)
in ()
7 # Training set
8 for images_batch, targets_batch in train_dataset.batch(batch_size):
----> 9 train_step(images_batch, targets_batch)
10 template = '\r Batch {}/{}, Loss: {}, Accuracy: {}'
11 print(template.format()

---

InvalidArgumentError: Received a label value of 26 which is outside the valid range of [0, 4). Label values: 21 21 2 2 17 17 15 15 18 18 26 26 6 6 9 9 6 6 12 12 8 8 16 16 3 3 20 20 6 6 17 17
[[node sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits (defined at :7) ]] [Op:__inference_train_step_2751]

Errors may have originated from an input operation.
Input Source operations connected to node sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits:
conv3d_model/output/BiasAdd (defined at :31)

Function call stack:
train_step