kangxi9 / cnn_lstm

Action recognition by CNN and LSTM

Overview

This code is for action recognition using the two stream networks (rgb and optical flow). Each stream is constructed by CNN and LSTM. I implemented this code for practice because I have not written any deep learning code from scratch!

Setup

Environment

• Linux (I used Ubuntu 14.04)
• GPU (I used GTX TitanX x2)
• Keras + TensorFlow (I recommend to install Keras via docker)

Dataset

Download UCF-101 dataset, and set to arbitary directory.

Run

Convert video to RGB images or optical flows

Use scripts/extract_optical_flow.sh in TSN (ECCV2016).

Make train/val split file

Use scripts/build_file_list.sh in TSN (ECCV2016).

Train model and test it

You can train a model and test it.

bash run.sh

TODO

• Modify to accept the various length input sequences (Fixed number of frames are extracted from a sequence now)

Reference

CNN + LSTM

• Keras issue 401
• Keras issue 5527
• LRCN (CVPR2015)

Batch training using "fit_generator" method

• Hatena blog

Two stream CNN

• Two stream CNN (NIPS2014)