zjh1124 / bin

Blurry Video Frame Interpolation (CVPR20)

BIN (Blurry Video Frame Interpolation)

Project | Paper

Wang Shen, Wenbo Bao, Guangtao Zhai, Li Chen, Xiongkuo Min, and Zhiyong Gao

IEEE Conference on Computer Vision and Pattern Recognition, Seattle, CVPR 2020

Table of Contents

1. Introduction
2. Citation
3. Requirements and Dependencies
4. Installation
5. Testing Pre-trained Models
6. Downloading Results
7. Training New Models

Introduction

We propose a Blurry video frame INterpolation method to reduce motion blur and up-convert frame rate simultaneously. We provide videos here.

Citation

If you find the code and datasets useful in your research, please cite:

Frame interpolation for blurry video @inproceedings{BIN, author = {Shen, Wang and Bao, Wenbo and Zhai, Guangtao and Chen, Li and Min, Xiongkuo and Gao, Zhiyong}, title = {Blurry Video Frame Interpolation}, booktitle = {IEEE Conference on Computer Vision and Pattern Recognition}, year = {2020} }

Frame interpolation for normal video @inproceedings{DAIN, author = {Bao, Wenbo and Lai, Wei-Sheng and Ma, Chao and Zhang, Xiaoyun and Gao, Zhiyong and Yang, Ming-Hsuan}, title = {Depth-Aware Video Frame Interpolation}, booktitle = {IEEE Conference on Computer Vision and Pattern Recognition}, year = {2019} }

Frame interpolation MEMC architecture @article{MEMC-Net, title={MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement}, author={Bao, Wenbo and Lai, Wei-Sheng, and Zhang, Xiaoyun and Gao, Zhiyong and Yang, Ming-Hsuan}, journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, doi={10.1109/TPAMI.2019.2941941}, year={2018} }

Requirements and Dependencies

• Ubuntu (We test with Ubuntu = 16.04.5 LTS)
• Python (We test with Python = 3.6.8 in Anaconda3 = 4.1.1)
• Cuda & Cudnn (We test with Cuda = 10.0 and Cudnn = 7.4)
• PyTorch >= 1.0.0 (We test with Pytorch = 1.3.0)
• FFmpeg (We test with the static build version = ffmpeg-git-20190701-amd64-static)
• GCC (Compiling PyTorch 1.0.0 extension files (.c/.cu) requires gcc = 4.9.1 and nvcc = 10.0 compilers)
• NVIDIA GPU (We use RTX-2080 Ti with compute = 7.5)

Installation

Download repository:

$ git clone https://github.com/laomao0/BIN.git

Make Adobe240 blur dataset

1. Download the Adobe240 original videos http://www.cs.ubc.ca/labs/imager/tr/2017/DeepVideoDeblurring/DeepVideoDeblurring_Dataset_Original_High_FPS_Videos.zip

2. Then de-compress those videos into a folder: Adobe_240fps_original_high_fps_videos

The structure of the folder is as following:

Adobe_240fps_original_high_fps_videos -- 720p_240fps_1.mov
                                        -- 720p_240fps_2.mov
                                        -- 720p_240fps_3.mov
                                        -- ...

3. Make the Adobe240 blur dataset by averaging N frames.

We averaging 11 consecutive frames to synthesize 1 blur image.

For example, the frame indexs of a 240-fps video are 0 1 2 3 4 5 6 7 8 9 10 11 12... We average 0-11 frames to synthesize the blur frame 0, average 8-19 frames to synthesize the blur frame 1. The frame rate of synthesized blur video is 30-fps.

$ cd data_scripts/adobe240fps
$ ./create_dataset_blur_N_frames_average_Adobe.sh

If you do not want to create the training set, setting --enable_train to be 0.

4. Check the dataset

The script of step 3 will create the dataset at path specified at --dataset_folder It contains 7 folders, including full_sharp, test, test_blur, test_list, train, train_blur, train_list

full_sharp: contain all de-compressed frames, not used in this project.
test/train: contain the sharp test/train frams at 240-fps.
test_blur/train_blur: contain the blur test/train frames at 30-fps.
test_list/train_list: contain im_list files used for dataloader.

test/train structure:
                        folder_1 -- 00001.png 00002.png ....
                        folder_2 -- 00001.png 00002.png ....
test_blur/train_blur structure:
                        folder_1 -- 00017.png 00025.png ....
                        folder_2 -- 00017.png 00025.png ....

Testing Pre-trained Models

1. Download pretrained models,

   $ cd model_weights $ download the model for Adobe240 dataset https://drive.google.com/open?id=1KGu8bLcIHODGQKw8fZ4NCVB1VgSVepo9

2. run the script

   $ bash run_bin_stage4.sh

3. check the results

the logging file and images are saved at --output_path/60fps_test_results/adobe_stage4

test_summary.log records PSNR, SSIM of each video folders

Training New Models

If you want to train the model on our own data

$ cd BIN
$ bash train.sh

Contact

Wang Shen; Wenbo Bao;

License