zhangnyg / actionrecognition

Action Recognition based on Pose Estimation. Graduation Project for Bachelor Degree.

ActionRecognition

Action Recognition based on pose estimation.

Graduation Project for Bachelor Degree
By Kenessary Koishybay, Nauryzbek Razakhbergenov.
Mentor: Anara Sandygulova. Nazarbayev University

Introduction

Pose estimation algortihm is based on tensorflow implementation of Realtime Multi-Person Pose Estimation

Requirements:

• Python 2.7
• OpenCV3
• sklearn
• scipy
• imutils
• xgboost

Running

To run my code you need to type:
    python -B Main.py <input_video> <output_video>
Here, arguments <input_video> and <output_video> are optional, and default values can be seen in the code.

How it works

Pose Estimation

Pose estimation is the process of locating body key points.
Pose estimation problem is usually solved by training Deep Learning architectures with annotated datasets such as MPII or COCO
We didn't have computational power to train on these datasets. Thus, we tried pre-trained model mentioned at the beginning.
Architecture:

However, after looking that even prediction on that architecture takes too much time, we use Mobile Net in the final version. We use pose estimation for Detection and collecting coordinates (x,y) of body key-points.

Tracking

Note, that we decided to remove code concerning EWMA in the final version.

Activity Recognition

Single Data Sample Length:
290 = 2*14*10 (x,y coords of 14 body parts in 10 frames) + 10(indexes of each frame)

For every (N = 10)th frame:

1. Open pose features calculated for every tracked humans
2. This features is then added to the previous features of the tracks
3. If the length of resulting feature vectors of specific tracks is large enough, feature vectors will be converted to data samples
4. These data samples is then goes as input to the machine learning algorithm (XGBoost)
5. XGBoost classifies activity of each data sample as code.
6. Code is then decoded into Activity Labels

Training

If you wan't to train our activity recognition algorithm to increase accuracy, first you should extract suitable data from videos dataset.

Data Extraction

We extracted data samples from KTH dataset.
Code is in ExtractData folder.

Model Selection and Training

TODO