serycjon / mft

MFT: Long-Term Tracking of Every Pixel -- code for the WACV 2024 paper

MFT: Long-Term Tracking of Every Pixel

Project Page Official implementation of the MFT tracker from the paper:

Michal Neoral, Jonáš Šerých, Jiří Matas: "MFT: Long-Term Tracking of Every Pixel", WACV 2024

Please cite our paper, if you use any of this.

@inproceedings{neoral2024mft,
               title={{MFT}: Long-Term Tracking of Every Pixel},
               author={Neoral, Michal and {\v{S}}er{\'{y}}ch, Jon{\'{a}}{\v{s}} and Matas, Ji{\v{r}}{\'{i}}},
               journal={arXiv preprint arXiv:2305.12998},
               year={2023},
}

Install

Create and activate a new virtualenv:

# we have tested with python 3.7.4
python -m venv venv
source venv/bin/activate

Then install all the dependencies:

pip install torch numpy einops tqdm opencv-python scipy Pillow==9 matplotlib ipdb

Run the demo

Simply running:

python demo.py

should produce a demo_out directory with two visualizations.

Training

See train.org

License

The demo video in demo_in was extracted from youtube.

This work is licensed under the Attribution-NonCommercial-ShareAlike 4.0 International license. The MFT/RAFT directory contains a modified version of RAFT, which is licensed under BSD-3-Clause license. Our modifications (OcclusionAndUncertaintyBlock and its integration in raft.py) are licensed again under the Attribution-NonCommercial-ShareAlike 4.0 International.

Acknowledgments

This work was supported by Toyota Motor Europe, by the Grant Agency of the Czech Technical University in Prague, grant No. SGS23/173/OHK3/3T/13, and by the Research Center for Informatics project CZ.02.1.01/0.0/0.0/16_019/0000765 funded by OP VVV.

mft's Issues

Inquiry About Downloading the Optical Flow Occlusion PNG Section of Flything3D Dataset

I am currently using your project, which utilizes the Flything3D dataset, for my research. First and foremost, I would like to express my gratitude for your excellent work and contribution to this field.

However, I am experiencing some difficulty in downloading the optical_flow_occlusion_png section of the Flything3D dataset. Could you please provide the download link or instructions on how to access this part of the dataset?

Thank you very much for your assistance and for your outstanding project!

Is the model capable for real-time tracking?

I am curious that is it possible for MFT to work in online tracking setting?