raphaelmemmesheimer / sl-dml

Signal Level Deep Metric Learning for One-Shot Action Recognition

SL-DML: Signal Level Deep Metric Learning for Multimodal One-Shot Action Recognition

This repository contains the code for SL-DML, a metric learning approach for one-shot action recognition supporting multiple modalities. A pre-print can be found on arxiv.

Video Abstract

Video

Citation

@inproceedings{memmesheimer2021sl,
  title={SL-DML: Signal Level Deep Metric Learning for Multimodal One-Shot Action Recognition},
  author={Memmesheimer, Raphael and Theisen, Nick and Paulus, Dietrich},
  booktitle={2020 25th International Conference on Pattern Recognition (ICPR)},
  pages={4573--4580},
  year={2021},
  organization={IEEE}
}

Requirements

• pip install -r requirements.txt
• SL-DML is based on the pytorch-metric-learning library

Precalculated Representations

We provide precalculated representations for all conducted experiment splits:

• NTU RGB+D 120 One-Shot
• UTD-MHAD
• Simitate

Quick Start

git clone https://github.com/raphaelmemmesheimer/sl-dml
cd sl-dml
pip install -r requirements.txt
export DATASET_FOLDER="$(pwd)/data"
mkdir -p data/ntu/
wget https://agas.uni-koblenz.de/datasets/sl-dml/ntu_120_one_shot.zip
unzip ntu_120_one_shot.zip -d $DATASET_FOLDER/ntu/ntu_swap_axes_testswapaxes
python train.py dataset=ntu_swap_axis

when returning you have to set the dataset folder again:

export DATASET_FOLDER="$(pwd)/data"
python train.py dataset=ntu_swap_axis

Training

Note, the following commands require an environment variable $DATASET_FOLDER to be existing.

NTU 120 One-Shot

Training for the NTU 120 one-shot action recognition experiments can be executed like:

python train.py dataset=ntu_swap_axis

During development, we suggest using the classes A002, A008, A014, A020, A026, A032, A038, A044, A050, A056, A062, A068, A074, A080, A086, A092, A098, A104, A110, A116 as validation classes.

Simitate One-Shot

wget https://agas.uni-koblenz.de/datasets/sl-dml/simitate_one_shot.zip
mkdir -p data/simitate/
unzip simitate_one_shot.zip -d $DATASET_FOLDER/simitate/

to train then:

python train.py dataset=simitate_aux_010_val_004

You can run all experiments on the Simitate dataset as:

python train.py dataset=simitate_aux_010_val_004,simitate_aux_010_val_016,simitate_aux_014_val_004,simitate_aux_014_val_012,simitate_aux_018_val_004,simitate_aux_018_val_008,simitate_aux_022_val_004 -m

sl-dml's Issues

IndexError: Target 83 is out of bounds.

Thanks for a great codebase!

I try to execute "python train.py dataset=ntu_swap_axis -m".
But I have encountered IndexError: Target 83 is out of bounds.

Do you know?

How to get an image represent from an inertial or skeleton data sequence

I've read your paper and code and would like to train and test under other datasets, but can't seem to get image representations. I hope you can provide the source code or describe in detail how to do it. Much appreciated!

hydra-core and omegaconf version

Hi, thank you so much for your outstanding research work. would you please tell me about the specific version of hydra-core and omegaconf?

many thanks

ValueError: Input contains NaN

Hi, I can start to train. But after evaluating epoch 1.
I encountered error

I guess that it is caused by train loss=Nan.
Do you have any suggestions?