mosamdabhi / mbw

This work generates 2D and 3D landmark labels from videos with only two or three uncalibrated, handheld cameras moving in the wild. NeurIPS 2022.

MBW: Multiview Bootstrapping in the Wild (NeurIPS 2022)

Paper | Project page | MBW-Data | Pretrained models

Multiview Bootstrapping in the Wild (MBW) provides a powerful way of generating labeled data in the wild at scale. Thereby, it democratizes the domain of data collection to a plethora of machine learning-driven computer vision applications via its novel self-supervision technique.

 

Requirements (if using GPU)

• Tested in Pytorch 1.11, with CUDA 11.3

 

Setup

1. Create a conda environment and activate it.

   conda env create -f environment_<cpu/gpu>.yml (change the flag within <> based on the available system)
   conda activate mbw
   pip install opencv-python
   

2. Please do a clean install of the submodule robust_loss_pytorch:

   cd modules/helpers/robust_loss_pytorch
   pip install git+https://github.com/jonbarron/robust_loss_pytorch
   

3. Please do a clean install of the submodule torch_batch_svd: (if using GPU)

   cd modules/helpers/torch-batch-svd
   export CUDA_HOME=/your/cuda/home/directory/    
   python setup.py install
   

 

Data & Pre-trained models

1. Fetch the pre-trained flow and detector models from Zenodo using:

   zenodo_get 10.5281/zenodo.7054596
   unzip models.zip
   rm -rf models.zip && rm -rf md5sums.txt
   

2. Download the data from and unzip it in the data directory.

   zenodo_get 10.5281/zenodo.7058567
   unzip data.zip
   rm -rf data.zip && rm -rf md5sums.txt
   

   The final directory after retrieving pre-trained models and sample data should look like this:

   ${mbw}
    `-- data
        `-- Chimpanzee
            |-- annot/
            |-- images/
        
    `-- models
        |-- detector/
        |-- flow/
        |-- mvnrsfm/
   

 

Run unit tests

./scripts/unit_tests.sh

Training (Generate labels from MBW)

./scripts/train.sh

Evaluation and visualization

./scripts/eval.sh
./scripts/visualize.sh    

Demo (Jupyter notebook examples)

Please run demo.ipynb to play around and see visualizations of labels in the provided Jupyter notebook. We provide tools to visualize the 2D predictions (along with a confidence flag of Accept or Reject). The reconstructed 3D could also be visualized via an interactive plotly visualization tool.

Citation

If you use our code, dataset, or models in your research, please cite with:

@inproceedings{dabhi2022mbw,
	title={MBW: Multi-view Bootstrapping in the Wild},
	author={Dabhi, Mosam and Wang, Chaoyang and Clifford, Tim and Jeni, Laszlo and Fasel, Ian and Lucey, Simon},
	booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
	year={2022},
	ee = {https://openreview.net/forum?id=i1bFPSw42W0},
	organization={NeurIPS}
}


@inproceedings{dabhi2021mvnrsfm,
	title={High Fidelity 3D Reconstructions with Limited Physical Views},
	author={Dabhi, Mosam and Wang, Chaoyang and Saluja, Kunal and Jeni, Laszlo and Fasel, Ian and Lucey, Simon},
	booktitle={2021 International Conference on 3D Vision (3DV)},
	year={2021},
	ee = {https://ieeexplore.ieee.org/abstract/document/9665845},
	organization={IEEE}
}