ponimatkin / ssl-vos

[WACV 2023] A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation

A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation

This repository contains the official PyTorch implementation of the following paper.

A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation,
Georgy Ponimatkin, Nermin Samet, Yang Xiao, Yuming Du, Renaud Marlet, Vincent Lepetit,
WACV 2023. (arXiv pre-print, Project page)

Preparing the environment and data

To prepare the environment run the following commands:

conda env create --name ssl-vos python=3.8 pip
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge
pip install scikit-learn opencv-python fast_slic imageio matplotlib scikit-image easydict path.py

python setup.py build
python setup.py install

To download the required assets run bash get_raw_data.sh and bash download_models.sh. Data can be prepared by running the following script

python prepare_data.py

Please refer to ARFlow and RAFT repositories in order to prepare rest of your environment.

Running the pipeline

Our approach requires three steps to run produce segmentations. At first, extract DINO features and optical flows by via

python extract_dino_features.py --dataset davis
python generate_flow_arflow.py --dataset davis --step 1

The second step consists of extracting the initial eigenvectors, which can be done by

python generate_pic_eigenvectors.py --use-gpu --dataset davis

Global optimization then can be run via

python global_optimization.py --dataset davis

The masks can be generated from the obtained solution by running

python extract_masks.py --dataset davis --method name_of_the_folder_in_data/davis

Citation

If you use this code in your research, please cite the following paper:

G. Ponimatkin, N. Samet, Y. Xiao, Y. Du, R. Marlet and V. Lepetit "A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation", In IEEE Winter Conference on Applications of Computer Vision (WACV), 2023.

BibTeX entry:

@inproceedings{ponimatkin2023sslvos, 
title= {A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation}, 
author={G. {Ponimatkin} and N. {Samet} and Y. {Xiao} and Y. {Du} and R. {Marlet} and V. {Lepetit}}, 
booktitle={Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV)}, 
year={2023}} }

ssl-vos's Issues

Errors in `generate_flow_arflow.py`

HI! I just got to get interested in the paper.

Traceback (most recent call last):
  File "generate_flow_arflow.py", line 13, in <module>
    from config import DATA_PATH, MAX_FLOW_VAL
ImportError: cannot import name 'MAX_FLOW_VAL' from 'config'

Also the following import would fail.

from utils.flow_utils import quantize_flow 

RAFT command

Thanks for the amazing work!

It seems that the README only includes ARFlow command. What is the command to extract raft flow?

Seems that generate_flow_raft.py accepts a --model argument. What should this be?