humaticslab / seam-match-rcnn

Official code of paper: MovingFashion: a Benchmark for the Video-to-Shop Challenge

SEAM Match-RCNN

Official code of MovingFashion: a Benchmark for the Video-to-Shop Challenge paper

Installation

Requirements:

• Pytorch 1.5.1 or more recent, with cudatoolkit (10.2)
• torchvision
• tensorboard
• cocoapi
• OpenCV Python
• tqdm
• cython
• CUDA >= 10

Step-by-step installation

# first, make sure that your conda is setup properly with the right environment
# for that, check that `which conda`, `which pip` and `which python` points to the
# right path. From a clean conda env, this is what you need to do

conda create --name seam -y python=3
conda activate seam

pip install cython tqdm opencv-python

# follow PyTorch installation in https://pytorch.org/get-started/locally/
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

conda install tensorboard

export INSTALL_DIR=$PWD

# install pycocotools
cd $INSTALL_DIR
git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI
python setup.py build_ext install

# download SEAM
cd $INSTALL_DIR
git clone https://github.com/VIPS4/SEAM-Match-RCNN.git
cd SEAM-Match-RCNN
mkdir data
mkdir ckpt

unset INSTALL_DIR

Dataset

SEAM Match-RCNN has been trained and test on MovingFashion and DeepFashion2 datasets. Follow the instruction to download and extract the datasets.

We suggest to download the datasets inside the folder data.

MovingFashion

MovingFashion dataset is available for academic purposes here.

Deepfashion2

DeepFashion2 dataset is available here. You need fill in the form to get password for unzipping files.

Once the dataset will be extracted, use the reserved DeepFtoCoco.py script to convert the annotations in COCO format, specifying dataset path.

python DeepFtoCoco.py --path <dataset_root>

Training

We provide the scripts to train both Match-RCNN and SEAM Match-RCNN. Check the scripts for all the possible parameters.

Single GPU

#training of Match-RCNN
python train_matchrcnn.py --root_train <path_of_images_folder> --train_annots <json_path> --save_path <save_path> 

#training on movingfashion
python train_movingfashion.py --root <path_of_dataset_root> --train_annots <json_path> --test_annots <json_path> --pretrained_path <path_of_matchrcnn_model>


#training on multi-deepfashion2
python train_multiDF2.py --root <path_of_dataset_root> --train_annots <json_path> --test_annots <json_path> --pretrained_path <path_of_matchrcnn_model>

Multi GPU

We use internally torch.distributed.launch in order to launch multi-gpu training. This utility function from PyTorch spawns as many Python processes as the number of GPUs we want to use, and each Python process will only use a single GPU.

#training of Match-RCNN
python -m torch.distributed.launch --nproc_per_node=<NUM_GPUS> train_matchrcnn.py --root_train <path_of_images_folder> --train_annots <json_path> --save_path <save_path>

#training on movingfashion
python -m torch.distributed.launch --nproc_per_node=<NUM_GPUS> train_movingfashion.py --root <path_of_dataset_root> --train_annots <json_path> --test_annots <json_path> --pretrained_path <path_of_matchrcnn_model> 

#training on multi-deepfashion2
python -m torch.distributed.launch --nproc_per_node=<NUM_GPUS> train_multiDF2.py --root <path_of_dataset_root> --train_annots <json_path> --test_annots <json_path> --pretrained_path <path_of_matchrcnn_model> 

Pre-Trained models

It is possibile to start training using the MatchRCNN pre-trained model.

[MatchRCNN] Pre-trained model on Deepfashion2 is available to download here. This model can be used to start the training at the second phase (training directly SEAM Match-RCNN).

We suggest to download the model inside the folder ckpt.

Evaluation

To evaluate the models of SEAM Match-RCNN please use the following scripts.

#evaluation on movingfashion
python evaluate_movingfashion.py --root_test <path_of_dataset_root> --test_annots <json_path> --ckpt_path <checkpoint_path>


#evaluation on multi-deepfashion2
python evaluate_multiDF2.py --root_test <path_of_dataset_root> --test_annots <json_path> --ckpt_path <checkpoint_path>

Citation

@misc{godi2021movingfashion,
      title={MovingFashion: a Benchmark for the Video-to-Shop Challenge}, 
      author={Marco Godi and Christian Joppi and Geri Skenderi and Marco Cristani},
      year={2021},
      eprint={2110.02627},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

seam-match-rcnn's Issues

Evaluation error Missing keys

Hi,
When I am running the evaluation on the pretrained model it is failing with
'''
RuntimeError: Error(s) in loading state_dict for VideoMatchRCNN:
Missing key(s) in state_dict: "backbone.body.conv1.weight", "backbone.body.bn1.weight", "backbone.body.bn1.bias", "backbone.body.bn1.running_mean", "backbone.body.bn1.running_var", "backbone.body.layer1.0.conv1.weight", "backbone.body.layer1.0.bn1.weight", "backbone.body.layer1.0.bn1.bias", "backbone.body.layer1.0.bn1.running_mean", "backbone.body.layer1.0.bn1.running_var", "backbone.body.layer1.0.conv2.weight", "backbone.body.layer1.0.bn2.weight", "backbone.body.layer1.0.bn2.bias", "backbone.body.layer1.0.bn2.running_mean", "backbone.body.layer1.0.bn2.running_var", "backbone.body.layer1.0.conv3.weight", "backbone.body.layer1.0.bn3.weight", "backbone.body.layer1.0.bn3.bias", "backbone.body.layer1.0.bn3.running_mean", "backbone.body.layer1.0.bn3.running_var", "backbone.body.layer1.0.downsample.0.weight", "backbone.body.layer1.0.downsample.1.weight", roi_heads.temporal_aggregator.newnlb.phi.weight", "roi_heads.temporal_aggregator.newnlb.phi.bias", "roi_heads.temporal_aggregator.newnlb.concat_project.0.weight".
Unexpected key(s) in state_dict: "module.backbone.body.conv1.weight", "module.backbone.body.bn1.weight", "module.backbone.body.bn1.bias", "module.backbone.body.bn1.running_mean", "module.backbone.body.bn1.running_var", "module.backbone.body.layer1.0.conv1.weight", "module.backbone.body.layer1.0.bn1.weight", "module.backbone.body.layer1.0.bn1.bias", "module.backbone.body.layer1.0.bn1.running_mean", "module.roi_heads.match_predictor.conv_seq.4.weight", "module.roi_heads.match_predictor.conv_seq.4.bias", "module.roi_heads.match_predictor.conv_seq.6.weight", "module.roi_heads.match_predictor.conv_seq.6.bias", "module.roi_heads.match_predictor.linear.0.weight", "module.roi_heads.match_predictor.linear.0.bias", "module.roi_heads.match_predictor.linear.1.weight", "module.roi_heads.match_predictor.linear.1.bias", "module.roi_heads.match_predictor.linear.1.running_mean", "module.roi_heads.match_predictor.linear.1.running_var", "module.roi_heads.match_predictor.linear.1.num_batches_tracked", "module.roi_heads.match_predictor.last.weight", "module.roi_heads.match_predictor.last.bias".
'''

Basically all the keys are have module added to the name compared to the config i believe.

May i know if this is an issue with the pretrained model uploaded? if so can you share the corrected one.

Thanks and Regards

retrieval results

Hi, the retrieval results I reproduced seems inaccurate and I have no idea what the problem is.
I used the weights pre-trained on DeepFashion2, with 2 gpus for distributed training, and the evaluation result is as follows:

Top-1 Retrieval Accuracy: 0.0626
Top-5 Retrieval Accuracy: 0.1818
Top-10 Retrieval Accuracy: 0.2727
Top-20 Retrieval Accuracy: 0.3667

---

Top-1 Retrieval Accuracy Product Max: 0.2727
Top-5 Retrieval Accuracy Product Max: 0.5253
Top-10 Retrieval Accuracy Product Max: 0.6263
Top-20 Retrieval Accuracy Product Max: 0.7778

---

Top-1 Retrieval Accuracy Product Avg Desc: 0.1212
Top-5 Retrieval Accuracy Product Avg Desc: 0.3535
Top-10 Retrieval Accuracy Product Avg Desc: 0.4545
Top-20 Retrieval Accuracy Product Avg Desc: 0.5455

---

Top-1 Retrieval Accuracy Product Aggr Desc: 0.1818
Top-5 Retrieval Accuracy Product Aggr Desc: 0.4141
Top-10 Retrieval Accuracy Product Aggr Desc: 0.4646
Top-20 Retrieval Accuracy Product Aggr Desc: 0.5455

---

Top-1 Retrieval Accuracy Product Avg Dist: 0.1010
Top-5 Retrieval Accuracy Product Avg Dist: 0.3434
Top-10 Retrieval Accuracy Product Avg Dist: 0.4242
Top-20 Retrieval Accuracy Product Avg Dist: 0.5556

---

Top-1 Retrieval Accuracy Product Max Dist: 0.0606
Top-5 Retrieval Accuracy Product Max Dist: 0.2323
Top-10 Retrieval Accuracy Product Max Dist: 0.4444
Top-20 Retrieval Accuracy Product Max Dist: 0.5051

---

Top-1 Retrieval Accuracy Product Max Score: 0.1111
Top-5 Retrieval Accuracy Product Max Score: 0.2626
Top-10 Retrieval Accuracy Product Max Score: 0.3939
Top-20 Retrieval Accuracy Product Max Score: 0.5051

---

Regular ONLY
Top-1 Retrieval Accuracy: 0.0826
Top-5 Retrieval Accuracy: 0.2261
Top-10 Retrieval Accuracy: 0.3232
Top-20 Retrieval Accuracy: 0.4217

---

Top-1 Retrieval Accuracy Product Avg Desc: 0.1739
Top-5 Retrieval Accuracy Product Avg Desc: 0.4493
Top-10 Retrieval Accuracy Product Avg Desc: 0.5362
Top-20 Retrieval Accuracy Product Avg Desc: 0.6377

---

Top-1 Retrieval Accuracy Product Aggr Desc: 0.2174
Top-5 Retrieval Accuracy Product Aggr Desc: 0.5217
Top-10 Retrieval Accuracy Product Aggr Desc: 0.5652
Top-20 Retrieval Accuracy Product Aggr Desc: 0.6522

---

Top-1 Retrieval Accuracy Product Avg Dist: 0.1449
Top-5 Retrieval Accuracy Product Avg Dist: 0.4493
Top-10 Retrieval Accuracy Product Avg Dist: 0.5072
Top-20 Retrieval Accuracy Product Avg Dist: 0.6667

---

Top-1 Retrieval Accuracy Product Max Dist: 0.0870
Top-5 Retrieval Accuracy Product Max Dist: 0.2899
Top-10 Retrieval Accuracy Product Max Dist: 0.5362
Top-20 Retrieval Accuracy Product Max Dist: 0.6087

---

Top-1 Retrieval Accuracy Product Max Score: 0.1449
Top-5 Retrieval Accuracy Product Max Score: 0.3333
Top-10 Retrieval Accuracy Product Max Score: 0.4783
Top-20 Retrieval Accuracy Product Max Score: 0.5942

---

Hard ONLY
Top-1 Retrieval Accuracy: 0.0167
Top-5 Retrieval Accuracy: 0.0800
Top-10 Retrieval Accuracy: 0.1567
Top-20 Retrieval Accuracy: 0.2400

---

Top-1 Retrieval Accuracy Product Avg Desc: 0.0000
Top-5 Retrieval Accuracy Product Avg Desc: 0.1333
Top-10 Retrieval Accuracy Product Avg Desc: 0.2667
Top-20 Retrieval Accuracy Product Avg Desc: 0.3333

---

Top-1 Retrieval Accuracy Product Aggr Desc: 0.1000
Top-5 Retrieval Accuracy Product Aggr Desc: 0.1667
Top-10 Retrieval Accuracy Product Aggr Desc: 0.2333
Top-20 Retrieval Accuracy Product Aggr Desc: 0.3000

---

Top-1 Retrieval Accuracy Product Avg Dist: 0.0000
Top-5 Retrieval Accuracy Product Avg Dist: 0.1000
Top-10 Retrieval Accuracy Product Avg Dist: 0.2333
Top-20 Retrieval Accuracy Product Avg Dist: 0.3000

---

Top-1 Retrieval Accuracy Product Max Dist: 0.0000
Top-5 Retrieval Accuracy Product Max Dist: 0.1000
Top-10 Retrieval Accuracy Product Max Dist: 0.2333
Top-20 Retrieval Accuracy Product Max Dist: 0.2667

---

Top-1 Retrieval Accuracy Product Max Score: 0.0333
Top-5 Retrieval Accuracy Product Max Score: 0.1000
Top-10 Retrieval Accuracy Product Max Score: 0.2000
Top-20 Retrieval Accuracy Product Max Score: 0.3000

---

Rank median: 36.0; rank 1st quartile: 7.0; rank 3rd quartile: 140.25
Average Track Length: 9.292929292929292

use the reserved DeepFtoCoco.py script

Hello, when I run DeepFtoCoco.py using the MovingFashion dataset, it displays 0it [00:00,? It/s],
The MovingFashion dataset contains the imgs and videos folders, test.json, and train.json, respectively. How can I solve it?

A plug-and-play evaluation.

First of all, thanks for your open-source codes.

Then, I want to make a request. Currently, the evaluate_movingfashion.py was custom made just for the SEAM-Match-RCNN model. For example, the temporal_aggregator and the way to compute distance are SEAM-Match-RCNN specified.

For models other than the SEAM-Match-RCNN, it is difficult to understand this evaluation code for evaluating performances on the MovingFashion dataset.

It would be appreciated if you could provide an evaluation code that works for all models, e.g., an evaluation function that takes as input the similarity matrix and labels of query-gallery samples, and outputs the top-k performances.

AttributeError: 'TemporalRoIHeads' object has no attribute 'tracking_predictor'

Hi, Thank you for your code!
when I run train_movingfashion.py, it runs to a AttributeError:
'TemporalRoIHeads' object has no attribute 'tracking_predictor' at video_matchrcnn.py (line 327, in load_saved_matchrcnn)
How can I fix it?

The download link movingFashion not working :(

Regarding MovingFashion Dataset

The link for downloading the MovingFashion dataset is not working.