tencentarc / tvts

Turning to Video for Transcript Sorting

Turning to Video for Transcript Sorting

This repo contains the official implementations of the two papers:

1. Learning Transferable Spatiotemporal Representations from Natural Script Knowledge
2. TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale

News

• [2023.02] 🎉 TVTS is accepted to CVPR 2023.
• [2023.03] The official code of TVTS has been released.
• [2023.05] 🚀 TVTSv2 is comming out! Please refer to this link for details.
• [2023.08] The official code of TVTSv2 and the pre-trained models have been released. All zero-shot evaluations are available on a single GPU. We provide scripts for extracting your own video features. Try it now 😎!

Introduction

Quickstart

Folder v1 contains the official code of TVTS. See v1-README for details.

Folder v2 contains the official code of TVTSv2, an upgraded version of TVTS that produces powerful video representations for out-of-the-box usage. See v2-README for details.

Citation

If you find our work helps, please cite our paper.

@InProceedings{Zeng_2023_CVPR,
    author    = {Zeng, Ziyun and Ge, Yuying and Liu, Xihui and Chen, Bin and Luo, Ping and Xia, Shu-Tao and Ge, Yixiao},
    title     = {Learning Transferable Spatiotemporal Representations From Natural Script Knowledge},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {23079-23089}
}

@misc{zeng2023tvtsv2,
      title={TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale}, 
      author={Ziyun Zeng and Yixiao Ge and Zhan Tong and Xihui Liu and Shu-Tao Xia and Ying Shan},
      year={2023},
      eprint={2305.14173},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

tvts's Issues

Question about SSV2_MC

Thanks for the great job. One of my questions is how to organize SSV2_MC. The code shows that there are options and answer in the annotations, which are different from the original annotations.

TVTS/v2/data_loader/SSV2_mc_dataset.py

Lines 70 to 72 in 290a63e

	meta_arr = {'raw_captions': 'NULL', 'paths': rel_fp, 'dataset': self.dataset_name}
	data = {'video': final, 'text': sample['options'], 'label': int(sample['answer']), 'meta': meta_arr,
	'keep_ind': keep_ind}

Could you please share the annotation files for SSV2_MC?