This repository contains the code for TriDet: Temporal Action Detection with Relative Boundary Modeling paper, which has been accepted for CVPR2023. Our code is built upon the codebase from ActionFormer and Detectron2, and we would like to express our gratitude for their outstanding work.
To quickly get start with the model architecture, you can focus mainly on the following files:
libs/modeling/blocks.py
libs/modeling/backbones.py
libs/modeling/meta_archs.py
- 2023.3.16 We release the code and the pretrained checkpoints.
- 2023.3.14 The pre-print version of our paper is updated to Arxiv.
- 2023.2.28 Our paper has been accepted for CVPR2023.
-
Please ensure that you have installed PyTorch and CUDA. (This code requires PyTorch version >= 1.11)
-
Install the required packages by running the following command:
pip install pyyaml pandas h5py joblib matplotlib
- Install NMS
cd ./libs/utils
python setup.py install --user
cd ../..
- Done! We are ready to get start!
-
We adpot the feature for THUMOS14, ActivityNet and Epic-Kitchen datasets from ActionFormer repository (see here). To use these features, please download them from their link and unpack them into the
./data
folder. -
For the HACS dataset, we use the official I3D feature of the RGB stream and the SlowFast feautre from TCANet in our experiments. Please unpack the I3D feature into
./data/hacs/i3d_feature
and the SlowFast feature into./data/hacs/slowfast_feature
. We provide processed annotation json files for the I3D feature and the SlowFast feature in the./data/hacs/annotations
folder. -
The folder structure for
./data/hacs/i3d_feature
should be as follows:./data/hacs/i3d_feature | |───training/ │ └───xxx.npy │ └───... └───validation/ │ └───xxx.npy │ └───... └───testing/ │ └───xxx.npy │ └───...
-
The folder structure for
./data/hacs/slowfast_feature
should be as follows:./data/hacs/slowfast_feature | |───training/ │ └───xxx.pkl │ └───... └───validation/ │ └───xxx.pkl │ └───...
We provide a list of scripts that allow you to reproduce our results with just one click. These scripts are located in
the ./tools
folder and include:
- thumos_i3d_script.sh
- epic_noun_slowfast_script.sh
- epic_verb_slowfast_script.sh
- hacs_slowfast_script.sh
- ant_tsp_script.sh
To easily reproduce our results, simply run the following command:
bash SCRIPT_PATH GPU_NUM
For example, if you want to train and eval our model on THUMOS14 dataset using the first GPU on you machine, you can run:
bash tools/thumos_i3d_script.sh 0
The mean average precision (mAP) results for each dataset are:
Dataset | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | Avg |
---|---|---|---|---|---|---|
THUMOS14 | 83.62 | 80.07 | 72.94 | 62.35 | 47.35 | 69.27 |
Dataset | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | Avg |
---|---|---|---|---|---|---|
EPIC-KITCHEN-noun | 27.38 | 26.28 | 24.60 | 22.23 | 18.28 | 23.76 |
Dataset | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | Avg |
---|---|---|---|---|---|---|
EPIC-KITCHEN-verb | 28.72 | 27.57 | 26.19 | 24.26 | 20.83 | 25.51 |
Dataset | 0.5 | 0.75 | 0.95 | Avg |
---|---|---|---|---|
HACS | 56.90 | 39.33 | 11.24 | 38.69 |
Dataset | 0.5 | 0.75 | 0.95 | Avg |
---|---|---|---|---|
ActivityNet | 54.71 | 38.01 | 8.35 | 36.77 |
*There has been a slight improvement in the results of some datasets compared to those reported in the paper.
We offer pre-trained models for each dataset, which you can download the chechpoints from Google Drive. The command for test is
python eval.py ./configs/CONFIG_FILE PATH_TO_CHECKPOINT
If you have any questions about the code, feel free to contact shidingfeng at buaa dot edu dot cn.
If you find this work helpful, please consider citing our paper
@article{shi2023tridet,
title = {TriDet: Temporal Action Detection with Relative Boundary Modeling},
author = {Shi, Dingfeng and Zhong, Yujie and Cao, Qiong and Ma, Lin and Li, Jia and Tao, Dacheng},
journal = {arXiv preprint arXiv:2303.07347},
year = {2023}
}