raozhongyu / two-stream-cnn

Two Stream CNN implemented in Keras using in skeleton-based action recognition with dataset NTU RGB+D

Two-Stream-CNN implement in Keras

Two Stream CNN is proposed in SKELETON-BASED ACTION RECOGNITION WITH CONVOLUTIONAL NEURAL NETWORKS, which is used for skeleton-based action recognition. It maps a skeleton sequence to an image( coordinates x,y,z to image R,G,B ). And they specially designed skeleton transformer module to rearrange and select important skeleton joints automatically.

Requirments

• Python3
• Keras
• h5py
• matplotlib
• numpy

Network Architecture

The network mainly consists of four modules which are Skeleton Transformer, ConvNet, Feature Fusion and Classification. The inputs of two stream are raw data(x, y, z) and frame difference respectively. As show below :

Usage

1. function/data_generator.py : generate the inputs numpy array of two stream

2. layers/transformer : the layer of Skeleton Transformer implement in Keras

3. network/ : the fold has four flies with different feature fusion way

Result

model	accuracy(cs)
base line	83.2%
my model	80.7%

Introduce attention mechanism to Skeleton Transformer module. Then, the accurancy can reach at 82.1%.

Contact

If you have any questions, please feel free to contact me.
Duohan Liang ([email protected])