An offical code repository for our paper
-
One-stage Video Instance Segmentation: from Frame-in Frame-out to Clip-in Clip-out
- [15/03/2022] Release code with pre-trained models on Github and paper on arxiv
-
Clone this repository
git clone https://github.com/MinghanLi/CiCo.git cd CiCo
-
Set up the environment using Anaconda:
conda env create -f environment.yml conda activate cico-env
-
According to your Cuda and pytorch version to install mmcv or mmcv-full from here.
# An example pytorch 1.10 and cuda 11.3 with mmcv version 1.4.2 pip install mmcv-full==1.4.2 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html
-
Compile a customized COCO API for YouTubeVIS dataset from here
pip install "git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI" git clone https://github.com/youtubevos/cocoapi cd cocoapi/PythonAPI # To compile and install locally python setup.py build_ext --inplace # To install library to Python site-packages python setup.py build_ext install
-
Datasets: If you'd like to train or test CiCo, please download the datasets from the official web: YTVIS2019, YTVIS2021 and OVIS, then update your data path in
configs/_base_/datasets/vis.py
cd CiCo vim configs/_base_/datasets/vis.py
-
Download CoCo-pretrained models with ResNet or Swin transformer backbones from here. To train, please put them in a directory, such as
outputs/pretrained_models/
.
# add --display to display instance masks
python eval.py --trained_model=path/to/your/trained/models.pth --NUM_CLIP_FRAMES=3 --overlap_frames=1
# Train CiCo with a 3-frame clip on 4 GPUs
CUDA_VISIBLE_DEVICES=0,1,2,3 python train.py --config=configs/CiCo/cico_yolact_r50_yt19.py --NUM_CLIP_FRAMES=3
--IMS_PER_BATCH=6 --LR=0.001 --backbone_dir=outputs/pretrained_models/
Backbone | FCN | mAP | Trained models | Results |
---|---|---|---|---|
R50 | Yolact | 37.1 | cico_yolact_r50_yt19_f3.pth | stdout.txt |
R50 | CondInst | 37.3 | cico_CondInst_r50_yt19_f3.pth | stdout.txt |
R101 | Yolact | 39.6 | ||
R101 | CondInst | 39.6 | cico_CondInst_r101_yt19_f3.pth | stdout.txt |
Swin-tiny | Yolact | 41.8 | stdout.txt | |
Swin-tiny | CondInst | 41.4 | cico_CondInst_swint_yt19_f3.pth | stdout.txt |
Backbone | FCN | mAP | Weights | Results |
---|---|---|---|---|
R50 | Yolact | 35.2 | cico_yolact_r50_yt21_f3.pth | stdout.txt |
R50 | CondInst | 35.4 | cico_CondInst_r50_yt21_f3.pth | stdout.txt |
R101 | Yolact | 36.5 | cico_yolact_r101_yt21_f3.pth | stdout.txt |
R101 | CondInst | 36.7 | cico_CondInst_r101_yt21_f3.pth | stdout.txt |
Swin-tiny | Yolact | 38.0 | ||
Swin-tiny | CondInst | 39.1 | cico_CondInst_swint_yt21_f3.pth | stdout.txt |
Backbone | FCN | mAP | Weights | Results |
---|---|---|---|---|
R50 | Yolact | 17.4 | cico_yolact_r50_ovis_f3.pth | stdout.txt |
R50 | CondInst | 18.0 | ||
R101 | Yolact | 19.1 | cico_yolact_r101_ovis_f3.pth | stdout.txt |
R101 | CondInst | 20.4 | cico_condinst_r101_ovis_f3.pth | stdout.txt |
Swin-tiny | Yolact | 18.0 | ||
Swin-tiny | CondInst | 18.2 |