This repository maintains the official implementation of the paper UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection by Ye Liu, Siyuan Li, Yang Wu, Chang Wen Chen, Ying Shan, and Xiaohu Qie, which has been accepted by CVPR 2022.
Please refer to the following environmental settings that we use. You may install these packages by yourself if you meet any problem during automatic installation.
- CUDA 11.5.0
- CUDNN 8.3.2.44
- Python 3.10.0
- PyTorch 1.11.0
- NNCore 0.3.6
- Clone the repository from GitHub.
git clone https://github.com/TencentARC/UMT.git
cd UMT
- Install dependencies.
pip install -r requirements.txt
- Download and extract the datasets.
- Prepare the files in the following structure.
UMT
├── configs
├── datasets
├── models
├── tools
├── data
│ ├── qvhighlights
│ │ ├── *features
│ │ ├── highlight_{train,val,test}_release.jsonl
│ │ └── subs_train.jsonl
│ ├── charades
│ │ ├── *features
│ │ └── charades_sta_{train,test}.txt
│ ├── youtube
│ │ ├── *features
│ │ └── youtube_anno.json
│ └── tvsum
│ ├── *features
│ └── tvsum_anno.json
├── README.md
├── setup.cfg
└── ···
Run the following command to train a model using a specified config.
# Single GPU
python tools/launch.py ${path-to-config}
# Multiple GPUs
torchrun --nproc_per_node=${num-gpus} tools/launch.py ${path-to-config}Run the following command to test a model and evaluate results.
python tools/launch.py ${path-to-config} --checkpoint ${path-to-checkpoint} --eval
Run the following command to pre-train a model using ASR captions on QVHighlights.
torchrun --nproc_per_node=4 tools/launch.py configs/qvhighlights/umt_base_pretrain_100e_asr.py
We provide multiple pre-trained models and training logs here. All the models are trained with a single NVIDIA Tesla V100-FHHL-16GB GPU and are evaluated using the default metrics of the datasets.
| Dataset | Model | Type | MR mAP | HD mAP | Download | ||
|---|---|---|---|---|---|---|---|
| [email protected] | [email protected] | [email protected] | [email protected] | ||||
| QVHighlights | UMT-B | — | 38.59 | 39.85 | model | metrics | ||
| UMT-B | w/ PT | 39.26 | 40.10 | model | metrics | |||
| Charades-STA | UMT-B | V + A | 48.31 | 29.25 | 88.79 | 56.08 | model | metrics |
| UMT-B | V + O | 49.35 | 26.16 | 89.41 | 54.95 | model | metrics | |
|
YouTube Highlights |
UMT-S | Dog | — | 65.93 | model | metrics | ||
| UMT-S | Gymnastics | — | 75.20 | model | metrics | |||
| UMT-S | Parkour | — | 81.64 | model | metrics | |||
| UMT-S | Skating | — | 71.81 | model | metrics | |||
| UMT-S | Skiing | — | 72.27 | model | metrics | |||
| UMT-S | Surfing | — | 82.71 | model | metrics | |||
| TVSum | UMT-S | VT | — | 87.54 | model | metrics | ||
| UMT-S | VU | — | 81.51 | model | metrics | |||
| UMT-S | GA | — | 88.22 | model | metrics | |||
| UMT-S | MS | — | 78.81 | model | metrics | |||
| UMT-S | PK | — | 81.42 | model | metrics | |||
| UMT-S | PR | — | 86.96 | model | metrics | |||
| UMT-S | FM | — | 75.96 | model | metrics | |||
| UMT-S | BK | — | 86.89 | model | metrics | |||
| UMT-S | BT | — | 84.42 | model | metrics | |||
| UMT-S | DS | — | 79.63 | model | metrics | |||
Here, w/ PT means initializing the model using pre-trained weights on ASR captions. V, A, and O indicate video, audio, and optical flow, respectively.
If you find this project useful for your research, please kindly cite our paper.
@inproceedings{liu2022umt,
title={UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection},
author={Liu, Ye and Li, Siyuan and Wu, Yang and Chen, Chang Wen and Shan, Ying and Qie, Xiaohu},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
pages={3042--3051},
year={2022}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent