xvjiarui / vfs

Rethinking Self-Supervised Correspondence Learning: A Video Frame-level Similarity Perspective, in ICCV 2021 (Oral)

Rethinking Self-Supervised Correspondence Learning: A Video Frame-level Similarity Perspective

This repository is the official implementation for VFS introduced in the paper:

Rethinking Self-Supervised Correspondence Learning: A Video Frame-level Similarity Perspective
Jiarui Xu, Xiaolong Wang
ICCV 2021 (Oral)

The project page with video is at https://jerryxu.net/VFS/.

Citation

If you find our work useful in your research, please cite:

@article{xu2021rethinking,
  title={Rethinking Self-Supervised Correspondence Learning: A Video Frame-level Similarity Perspective},
  author={Xu, Jiarui and Wang, Xiaolong},
  journal={arXiv preprint arXiv:2103.17263},
  year={2021}
}

Environmental Setup

• Python 3.7
• PyTorch 1.6-1.8
• mmaction2
• davis2017-evaluation
• got10k

The codebase is implemented based on the awesome MMAction2, please follow the install instruction of MMAction2 to setup the environment.

Quick start full script:

conda create -n vfs python=3.7 -y
conda activate vfs
conda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=11.1 -c pytorch -c conda-forge
pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.8.0/index.html
# install customized evaluation API for DAVIS
pip install git+https://github.com/xvjiarui/davis2017-evaluation
# install evaluation API for OTB
pip install got10k

# install VFS
git clone https://github.com/xvjiarui/VFS/
cd VFS
pip install -e .

We also provide the Dockerfile under docker/ folder.

The code is developed and tested based on PyTorch 1.6-1.8. It also runs smoothly with PyTorch 1.9 but the accuracy is slightly worse for OTB evaluation. Please feel free to open a PR if you find the reason.

Model Zoo

Fine-grained correspondence

Backbone	Config	J&F-Mean	J-Mean	F-Mean	Download	Inference cmd
ResNet-18	cfg	66.7	64.0	69.5	pretrain ckpt	cmd ./tools/dist_test.sh configs/r18_nc_sgd_cos_100e_r2_1xNx8_k400.py https://github.com/xvjiarui/VFS/releases/download/v0.1-rc1/r18_nc_sgd_cos_100e_r2_1xNx8_k400-db1a4c0d.pth 1 --eval davis --options test_cfg.save_np=True
ResNet-50	cfg	69.5	67.0	72.0	pretrain ckpt	cmd ./tools/dist_test.sh configs/r50_nc_sgd_cos_100e_r5_1xNx2_k400.py https://github.com/xvjiarui/VFS/releases/download/v0.1-rc1/r50_nc_sgd_cos_100e_r5_1xNx2_k400-d7ce3ad0.pth 1 --eval davis --options test_cfg.save_np=True

Note: We report the accuracy of the last block in res4, to evaluate all blocks, please pass --options test_cfg.all_blocks=True. The reproduced performance in this repo is slightly higher than reported in the paper.

Object-level correspondence

Backbone	Config	Precision	Success	Download	Inference cmd
ResNet-18	cfg	70.0	52.3	tracking ckpt	cmd python projects/siamfc-pytorch/train_siamfc.py configs/r18_sgd_cos_100e_r2_1xNx8_k400.py --checkpoint https://github.com/xvjiarui/VFS/releases/download/v0.1-rc1/r18_sgd_cos_100e_r2_1xNx8_k400-e3b6a4bc.pth
ResNet-50	cfg	73.9	52.5	tracking ckpt	cmd python projects/siamfc-pytorch/train_siamfc.py configs/r50_sgd_cos_100e_r5_1xNx2_k400.py --checkpoint https://github.com/xvjiarui/VFS/releases/download/v0.1-rc1/r50_sgd_cos_100e_r2_1xNx2_k400-b7fb2a38.pth --options out_scale=0.00001 out_channels=2048

Note: We fine-tune an extra linear layer. The reproduced performance in this repo is slightly higher than reported in the paper.

Data Preparation

We use Kinetics-400 for self-supervised correspondence pretraining.

The fine-grained correspondence is evaluated on DAVIS2017 w/o any fine-tuning.

The object-level correspondence is evaluated on OTB-100 under linear probing setting (fine-tuning an extra linear layer).

The overall file structure is as followed:

vfs
├── mmaction
├── tools
├── configs
├── data
│   ├── kinetics400
│   │   ├── videos_train
│   │   │   ├── kinetics400_train_list_videos.txt
│   │   │   ├── train
│   │   │   │   ├── abseiling/
│   │   │   │   ├── air_drumming/
│   │   │   │   ├── ...
│   │   │   │   ├── yoga/
│   │   │   │   ├── zumba/
│   ├── davis
│   │   ├── DAVIS
│   │   │   ├── Annotations
│   │   │   │   ├── 480p
│   │   │   │   │   ├── bike-packing/
│   │   │   │   │   ├── ...
│   │   │   │   │   ├── soapbox/
│   │   │   ├── ImageSets
│   │   │   │   ├── 2017/
│   │   │   │   ├── davis2017_val_list_rawframes.txt
│   │   │   ├── JPEGImages
│   │   │   │   ├── 480p
│   │   │   │   │   ├── bike-packing/
│   │   │   │   │   ├── ...
│   │   │   │   │   ├── soapbox/
│   ├── otb
│   │   ├── Basketball/
│   │   ├── ...
│   │   ├── Woman/
│   ├── GOT-10k
│   │   ├── train
│   │   │   ├── GOT-10k_Train_000001/
│   │   │   ├── ...
│   │   │   ├── GOT-10k_Train_009335/

The instructions for preparing each dataset are as followed.

Kinetics-400

Please follow the documentation here to prepare the Kinetics-400. The dataset could be downloaded from kinetics-dataset.

DAVIS2017

DAVIS2017 dataset could be downloaded from the official website. We use the 480p validation set for evaluation.

# download data
wget https://data.vision.ee.ethz.ch/csergi/share/davis/DAVIS-2017-trainval-480p.zip
# download filelist
wget https://github.com/xvjiarui/VFS/releases/download/v0.1-rc1/davis2017_val_list_rawframes.txt

Then please unzip and place them according to the file structure above.

OTB-100

The OTB-100 frames and annotations will be downloaded automatically.

GOT-10k

GOT-10k dataset could be downloaded from the official website.

Then please unzip and place them according to the file structure above.

Run Experiments

Pretrain

./tools/dist_train.sh ${CONFIG} ${GPUS}

We use 2 and 8 GPUs for ResNet-18 and ResNet-50 models respectively.

Inference

To run the following inference and evaluation, we need to convert the pretrained checkpoint into the same format as torchvision ResNet.

python tools/convert_weights/convert_to_pretrained.py ${PRETRAIN_CHECKPOINT} ${BACKBONE_WEIGHT}

Evaluate fine-grained correspondence on DAVIS2017

./tools/dist_test.sh ${CONFIG} ${BACKBONE_WEIGHT} ${GPUS}  --eval davis

You may pass --options test_cfg.save_np=True to save memory.

Inference cmd examples:

# testing r18 model
./tools/dist_test.sh configs/r18_nc_sgd_cos_100e_r2_1xNx8_k400.py https://github.com/xvjiarui/VFS/releases/download/v0.1-rc1/r18_nc_sgd_cos_100e_r2_1xNx8_k400-db1a4c0d.pth 1  --eval davis --options test_cfg.save_np=True
# testing r50 model
./tools/dist_test.sh configs/r50_nc_sgd_cos_100e_r5_1xNx2_k400.py https://github.com/xvjiarui/VFS/releases/download/v0.1-rc1/r50_nc_sgd_cos_100e_r5_1xNx2_k400-d7ce3ad0.pth 1  --eval davis --options test_cfg.save_np=True

Evaluate object-level correspondence

ResNet-18:

 python projects/siamfc-pytorch/train_siamfc.py ${CONFIG} --pretrained ${BACKBONE_WEIGHT}

ResNet-50:

 python projects/siamfc-pytorch/train_siamfc.py ${CONFIG} --pretrained ${BACKBONE_WEIGHT} --options out_scale=0.00001 out_channels=2048

The results will be saved in work_dirs/${CONFIG}/siamfc.

To inference with provided tracking checkpoints:

 python projects/siamfc-pytorch/train_siamfc.py ${CONFIG} --checkpoint ${TRACKING_CHECKPOINT}

Inference cmd examples:

# testing r18 model
python projects/siamfc-pytorch/train_siamfc.py configs/r18_sgd_cos_100e_r2_1xNx8_k400.py --checkpoint https://github.com/xvjiarui/VFS/releases/download/v0.1-rc1/r18_sgd_cos_100e_r2_1xNx8_k400-e3b6a4bc.pth
# testing r50 model
python projects/siamfc-pytorch/train_siamfc.py configs/r50_sgd_cos_100e_r5_1xNx2_k400.py --checkpoint https://github.com/xvjiarui/VFS/releases/download/v0.1-rc1/r50_sgd_cos_100e_r5_1xNx2_k400-b7fb2a38.pth --options out_scale=0.00001 out_channels=2048

Acknowledgements

The codebase is based on MMAction2. The fine-grained correspondence inference and evaluation follows TimeCycle, UVC and videowalk. The object-level correspondence inference and evaluation is based on SiamFC-PyTorch and vince.

Thank you all for the great open source repositories!

vfs's Issues

Training Logs on Resnet18 Backbone

Could you please share Training Logs on the ResNet18 backbone?

Regards

Failed to reproduce the result (finally solved, due to bugs in dataset setup)

Hi, I've tried to reproduce the result on DAVIS with ResNet18 backbone with the script as follows.

./tools/dist_train.sh configs/r18_nc_sgd_cos_100e_r2_1xNx8_k400.py 2

The official ckpt that you released is 66.73% J&F. However, for my retrained model, I got only ~60.86% J&F for the final checkpoint (epoch=100). The detailed evaluation result on DAVIS is as follows. So I am not sure what's going wrong.

What's more, I would like to ask you about the deviation of results. Thanks.

Results copypaste 60.86
J&F-Mean: 0.6086
J-Mean: 0.5830
J-Recall: 0.6685
J-Decay: 0.2298
F-Mean: 0.6343
F-Recall: 0.7311
F-Decay: 0.2853

The split txt used for JHMDB and VIP

hi,

May I ask where did you get the split txt file for the evaluation of JHMDB and VIP dataset?
I didn't find these two split file in the videowalk repo.
More concretely, I am confused about the split of validation videos (or you just used all the videos? ).
I will be appreciated if you would like to share the code for the evaluation of these two datasets.

best,

Issues relating to using ann_file_train for Kinetics400 dataset

Hi,

When I train the model, I face the following issues either use "kinetics400_train_list_videos.txt" or "kinetics400_train_list_videos.js
on" :
1)txt file error:
Traceback (most recent call last):
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/site-packages/mmcv/utils/registry.py", line 51, in build_from_cfg
return obj_cls(**args)
File "/home/hk20/VFS/mmaction/datasets/video_dataset.py", line 43, in init
super().init(ann_file, pipeline, start_index=start_index, **kwargs)
File "/home/hk20/VFS/mmaction/datasets/base.py", line 67, in init
self.video_infos = self.load_annotations()
File "/home/hk20/VFS/mmaction/datasets/video_dataset.py", line 61, in load_annotations
filename, label = line_split
ValueError: not enough values to unpack (expected 2, got 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./tools/train.py", line 181, in
main()
File "./tools/train.py", line 156, in main
datasets = [build_dataset(cfg.data.train)]
File "/home/hk20/VFS/mmaction/datasets/builder.py", line 37, in build_dataset
build_dataset(cfg['dataset'], default_args), cfg['times'])
File "/home/hk20/VFS/mmaction/datasets/builder.py", line 39, in build_dataset
dataset = build_from_cfg(cfg, DATASETS, default_args)
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/site-packages/mmcv/utils/registry.py", line 54, in build_from_cfg
raise type(e)(f'{obj_cls.name}: {e}')
ValueError: VideoDataset: not enough values to unpack (expected 2, got 1)
Killing subprocess 2017741
Traceback (most recent call last):
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/site-packages/torch/distributed/launch.py", line 340, in
main()
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/site-packages/torch/distributed/launch.py", line 326, in main
sigkill_handler(signal.SIGTERM, None) # not coming back
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/hk20/anaconda3/envs/vfs/bin/python', '-u', './tools/train.py', '--local_rank=0', 'configs/r18_nc_sgd_cos_100e_r2_1xNx8_k400.py', '--launcher', 'pytorch']' returned non-zero exit status 1.

2)json file error
Traceback (most recent call last):
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/site-packages/mmcv/utils/registry.py", line 51, in build_from_cfg
return obj_cls(**args)
File "/home/hk20/VFS/mmaction/datasets/video_dataset.py", line 43, in init
super().init(ann_file, pipeline, start_index=start_index, **kwargs)
File "/home/hk20/VFS/mmaction/datasets/base.py", line 67, in init
self.video_infos = self.load_annotations()
File "/home/hk20/VFS/mmaction/datasets/video_dataset.py", line 48, in load_annotations
return self.load_json_annotations()
File "/home/hk20/VFS/mmaction/datasets/base.py", line 80, in load_json_annotations
path_key = 'frame_dir' if 'frame_dir' in video_infos[0] else 'filename'
KeyError: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./tools/train.py", line 181, in
main()
File "./tools/train.py", line 156, in main
datasets = [build_dataset(cfg.data.train)]
File "/home/hk20/VFS/mmaction/datasets/builder.py", line 37, in build_dataset
build_dataset(cfg['dataset'], default_args), cfg['times'])
File "/home/hk20/VFS/mmaction/datasets/builder.py", line 39, in build_dataset
dataset = build_from_cfg(cfg, DATASETS, default_args)
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/site-packages/mmcv/utils/registry.py", line 54, in build_from_cfg
raise type(e)(f'{obj_cls.name}: {e}')
KeyError: 'VideoDataset: 0'
Killing subprocess 2017835
Traceback (most recent call last):
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/site-packages/torch/distributed/launch.py", line 340, in
main()
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/site-packages/torch/distributed/launch.py", line 326, in main
sigkill_handler(signal.SIGTERM, None) # not coming back
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/hk20/anaconda3/envs/vfs/bin/python', '-u', './tools/train.py', '--local_rank=0', 'configs/r18_nc_sgd_cos_100e_r2_1xNx8_k400.py', '--launcher', 'pytorch']' returned non-zero exit status 1.

I also attached the txt file that I am using, any idea how to solve this issue?
kinetics400_train_list_videos.txt

Welcome update to OpenMMLab 2.0

Welcome update to OpenMMLab 2.0

I am Vansin, the technical operator of OpenMMLab. In September of last year, we announced the release of OpenMMLab 2.0 at the World Artificial Intelligence Conference in Shanghai. We invite you to upgrade your algorithm library to OpenMMLab 2.0 using MMEngine, which can be used for both research and commercial purposes. If you have any questions, please feel free to join us on the OpenMMLab Discord at https://discord.gg/amFNsyUBvm or add me on WeChat (van-sin) and I will invite you to the OpenMMLab WeChat group.

Here are the OpenMMLab 2.0 repos branches:

	OpenMMLab 1.0 branch	OpenMMLab 2.0 branch
MMEngine		0.x
MMCV	1.x	2.x
MMDetection	0.x 、1.x、2.x	3.x
MMAction2	0.x	1.x
MMClassification	0.x	1.x
MMSegmentation	0.x	1.x
MMDetection3D	0.x	1.x
MMEditing	0.x	1.x
MMPose	0.x	1.x
MMDeploy	0.x	1.x
MMTracking	0.x	1.x
MMOCR	0.x	1.x
MMRazor	0.x	1.x
MMSelfSup	0.x	1.x
MMRotate	1.x	1.x
MMYOLO		0.x

Attention: please create a new virtual environment for OpenMMLab 2.0.

request for resnet pretrained on kinetics400

Hi, thx for the great work and open source repository!
Due to the poor computing resource, I can not train the resnet18 and resnet50 on kinetics400. Could you provide the pretrained model?

ModuleNotFoundError: No module named 'mmaction.version'

Hi,
I am trying to run pretraining through "./tools/dist_train.sh ${CONFIG} ${GPUS}", however I get the following error:

Traceback (most recent call last):
File "./tools/train.py", line 13, in
from mmaction import version
File "/home/hk20/VFS/mmaction/init.py", line 1, in
from .version import version, short_version
ModuleNotFoundError: No module named 'mmaction.version'
Killing subprocess 1976193
Traceback (most recent call last):
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/site-packages/torch/distributed/launch.py", line 340, in
main()
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/site-packages/torch/distributed/launch.py", line 326, in main
sigkill_handler(signal.SIGTERM, None) # not coming back
File "/home/hk20/anaconda3/envs/vfs/lib/python3.7/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/hk20/anaconda3/envs/vfs/bin/python', '-u', './tools/train.py', '--local_rank=0', 'configs/r18_nc_sgd_cos_100e_r2_1xNx8_k400.py', '--launcher', 'pytorch']' returned non-zero exit status 1.

Any idea how to solve this issue?

Pretrained model for the VIP dataset

Thanks for the code !!!
Could you also release the pretrained model for the VIP dataset? That would be very helpful!

How to run a demo to a new videos or bunch of frames?

Hi, to run the code on a similar costume video?
Do we need to prepare labels for all the frames, or is only the first frame enough?

Structuring Kinetics400 dataset

Hello,

Thanks for sharing your code.
I am trying to replicate your code and followed your instructions in the repo. However, after downloading Kinetics400 dataset, its file structure is quite different from what you suggested for the dataset in the file structure although I followed the link that you have suggested for the dataset downloading. I just wondering if you have any suggestions for that issue?

How long does it take to train for 100 epochs?

Hi,
May I ask how long does it take to train for 100 epochs and 500epochs on Kinetics, and how many GPUs did you use?

Request the precomputed results on DAVIS17

Hi, I'd like to request the precomputed masks on DAVIS17 with ResNet18 and ResNet50. Thanks!

Kinetics400 videos size and training times

Hi, I'm trying to reproduce the training of the model using a ResNet-18 on two Quadro P6000 (24GB VRAM each) and the estimated duration of the training for 100 epochs is over 15 days.
I think the long training times are due to the high resolution of the videos of kinetics. So, my questions are:

• did you use a resized version of the dataset (256px on the shorter side)?
• If yes, did you resize the videos using the script provided in mmaction?

Thank you in advance for your answer

Finding out frame similarity matrix between two video frames

Hello,

I was wondering if your code can calculate the frame similarity matrix between two video frames? If yes, which part of your code can do this?Thanks

Testing on DAVIS gives error from rawframe_dataset.py

Hi,

I'm getting error from rawframe_dataset.py.

Platform

Testing this on Google Colab pro VM High RAM (25GB), P100 GPU.
Pytorch 1.9.0
cuda 11.1
python 3.7.12
torchvision 0.10.0

Script

!pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.9.0/index.html

!pip install git+https://github.com/xvjiarui/davis2017-evaluation

!pip install got10k

!git clone https://github.com/xvjiarui/VFS/
!mv VFS/* ./
!pip install -e .
!wget https://data.vision.ee.ethz.ch/csergi/share/davis/DAVIS-2017-trainval-480p.zip
!mkdir ./data
!mkdir ./data/davis
!unzip -q DAVIS-2017-trainval-480p.zip -d ./data/davis/
!wget https://github.com/xvjiarui/VFS/releases/download/v0.1-rc1/davis2017_val_list_rawframes.txt -o ./data/davis/DAVIS/ImageSets/davis2017_val_list_rawframes.txt

!./tools/dist_test.sh configs/r18_nc_sgd_cos_100e_r2_1xNx8_k400.py https://github.com/xvjiarui/VFS/releases/download/v0.1-rc1/r18_nc_sgd_cos_100e_r2_1xNx8_k400-db1a4c0d.pth 1 --eval davis --options test_cfg.save_np=True

Following are the details of error.

Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/mmcv/utils/registry.py", line 52, in build_from_cfg
return obj_cls(**args)
File "/content/mmaction/datasets/davis_dataset.py", line 53, in init
modality='RGB')
File "/content/mmaction/datasets/rawframe_dataset.py", line 93, in init
multi_class, num_classes, start_index, modality)
File "/content/mmaction/datasets/base.py", line 67, in init
self.video_infos = self.load_annotations()
File "/content/mmaction/datasets/rawframe_dataset.py", line 118, in load_annotations
video_info['total_frames'] = int(line_split[idx])
ValueError: invalid literal for int() with base 10: '06:58:52--'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./tools/test.py", line 166, in
main()
File "./tools/test.py", line 120, in main
dataset = build_dataset(cfg.data.test, dict(test_mode=True))
File "/content/mmaction/datasets/builder.py", line 39, in build_dataset
dataset = build_from_cfg(cfg, DATASETS, default_args)
File "/usr/local/lib/python3.7/dist-packages/mmcv/utils/registry.py", line 55, in build_from_cfg
raise type(e)(f'{obj_cls.name}: {e}')

the pretrained model

Hi:
Thanks for your great work.
I noticed you report the performance of several image-based contrastive methods. Recently I am doing some ablations with both your method and Simsiam. Due to the limited computing resources, I can hardly train with it.
If you still have these pre-trained weights (Simsiam and MoCo with ResNet18 trained on ImageNet ), could you please share these with me? This is of great help to me. Looking forward to your reply.
Thanks for your time.

Tuple object is not callable

Hello,

I am trying to reproduce the training for resnet18 with the following command:

python tools/train.py configs/r18_sgd_cos_100e_r2_1xNx8_k400.py

but I receive the following error within mmaction:

...
2022-03-31 11:09:23,556 - mmaction - INFO - workflow: [('train', 1)], max: 100 epochs
2022-03-31 11:09:23,556 - mmaction - INFO - Checkpoints will be saved to /home/smodi9/cs598/VFS/work_dirs/r18_sgd_cos_100e_r2_1xNx8_k400 by HardDiskBackend.
Traceback (most recent call last):
  File "tools/train.py", line 181, in <module>
    main()
  File "tools/train.py", line 177, in main
    meta=meta)
  File "/home/smodi9/cs598/VFS/mmaction/apis/train.py", line 119, in train_model
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/smodi9/miniconda3/envs/vfs/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/smodi9/miniconda3/envs/vfs/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 47, in train
    for i, data_batch in enumerate(self.data_loader):
  File "/home/smodi9/miniconda3/envs/vfs/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 517, in __next__
    data = self._next_data()
  File "/home/smodi9/miniconda3/envs/vfs/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1199, in _next_data
    return self._process_data(data)
  File "/home/smodi9/miniconda3/envs/vfs/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1225, in _process_data
    data.reraise()
  File "/home/smodi9/miniconda3/envs/vfs/lib/python3.7/site-packages/torch/_utils.py", line 429, in reraise
    raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/smodi9/miniconda3/envs/vfs/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/smodi9/miniconda3/envs/vfs/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/smodi9/miniconda3/envs/vfs/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/smodi9/cs598/VFS/mmaction/datasets/dataset_wrappers.py", line 26, in __getitem__
    return self.dataset[idx % self._ori_len]
  File "/home/smodi9/cs598/VFS/mmaction/datasets/base.py", line 139, in __getitem__
    data = self.prepare_train_frames(idx)
  File "/home/smodi9/cs598/VFS/mmaction/datasets/base.py", line 119, in prepare_train_frames
    return self.pipeline(results)
  File "/home/smodi9/cs598/VFS/mmaction/datasets/pipelines/compose.py", line 41, in __call__
    data = t(data)
  File "/home/smodi9/cs598/VFS/mmaction/datasets/pipelines/augmentations.py", line 1315, in __call__
    img = np.array(trans(Image.fromarray(img)))
TypeError: 'tuple' object is not callable

I followed the instructions in the installation README and I believe the issue is caused by _ColorJitter.get_params(...) which returns a tuple. I am on Pytorch 1.8 and Torchvision 0.9.0.

Any help would be greatly appreciated!

backbone.init_weights issue

Hi, when evaluating object-level correspondence on OTB100 using resnet18, I encounter the error following:

2021-12-05 16:23:08,218 - mmaction - INFO - Loading ../../checkpoints/resnet18/r18_sgd_cos_100e_r2_1xNx8_k400-99e2f7cd.pth as torchvision
2021-12-05 16:23:08,218 - mmaction - INFO - Use load_from_local loader
Traceback (most recent call last):
  File "/VFS/projects/siamfc-pytorch/train_siamfc.py", line 179, in <module>
    main()
  File "/VFS/projects/siamfc-pytorch/train_siamfc.py", line 114, in main
    tracker = TrackerSiamFC(cfg, logger)
  File "/VFS/projects/siamfc-pytorch/siamfc/siamfc_tracker_base.py", line 102, in __init__
    backbone.init_weights()
  File "/VFS/mmaction/models/backbones/resnet.py", line 534, in init_weights
    self.pretrained, strict=False, logger=logger)
  File "o/VFS/mmaction/models/backbones/resnet.py", line 513, in _load_torchvision_checkpoint
    original_conv_name, loaded_param_names)
  File "/VFS/mmaction/models/backbones/resnet.py", line 452, in _load_conv_params
    conv.weight.data.copy_(state_dict_tv[weight_tv_name])
KeyError: 'conv1.weight'

Process finished with exit code 1

I find that the checkpoint you provided does not match the cmd you given (python projects/siamfc-pytorch/train_siamfc.py configs/r18_sgd_cos_100e_r2_1xNx8_k400.py --checkpoint https://github.com/xvjiarui/VFS/releases/download/v0.1-rc1/r18_sgd_cos_100e_r2_1xNx8_k400-e3b6a4bc.pth).
And the url of the checkpoint is not exist.
So is this the main reason for this error? If not , should I change the _load_torchvision_checkpoint function in VFS\mmaction\models\backbones\resnet.py to meet the request?