esw0116 / dynavsr

DynaVSR: Dynamic Adaptive Blind VideoSuper-Resolution

DynaVSR

DynaVSR: Dynamic Adaptive Blind VideoSuper-Resolution

Suyoung Lee*, Myungsub Choi*, Kyoung Mu Lee

Seoul National University

This repository is the official implementation of DynaVSR, which is accepted in WACV2021.

Link: Paper(Arxiv) | Paper(CVFOpenAccess)

Directory Structure

project
│   README.md
└───dataset - make symbolic link of your data here
└───pretrained_models - put the downloadedpretrained model here
└───codes
│   └───data
│       │   common.py
│       │   data_sampler.py
│       │   old_kernel_generator.py
│       │   random_kernel_generator.py
│       └───baseline - for finetuning the VSR network
│       └───estimator - for training MFDN, SFDN
│       └───meta_learner - training maml network
│   └───data_scripts - miscellaneous scripts (same as EDVR)
│   └───metrics - metric calculation (same as EDVR)
│   └───models - model collections
│   └───options
│       └───test - ymls for testing the networks
│       └───train - ymls for training the networks
│       │   options.py
│   └───scripts
│   └───utils
|   calc_psnr_ssim.py - code for calculating psnrs and ssims for image sets
|   degradation_gen.py - code for generating the preset kernel
|   make_downscaled_images.py - code for making true slr images
|   make_slr_images.py - code for making slr images generated by MFDN, SFDN
|   train.py - code for training the VSR network
|   train_dynavsr.py  - code for training DynaVSR
|   train_mfdn.py - code for training MFDN, SFDN network
|   test_maml.py - code for testing DynaVSR
|   test_Vid4_REDS4_with_GT(_DUF, _TOF).py - code for testing baseline VSR network
|   run_downscaling.sh - scripts for generating LR, SLR images
|   run_visual.sh - scripts for testing DynaVSR
| ...

Dependencies

Current version is tested on:

• Ubuntu 18.04
• Python==3.7.7
• numpy==1.17
• PyTorch==1.3.1, torchvision==0.4.2, cudatoolkit==10.0
• tensorboard==1.14.0
• etc: pyyaml, opencv, scikit-image, pandas, imageio, tqdm

# Easy installation (using Anaconda environment)
conda env create -f environment.yml
conda activate dynavsr

Model

Dataset Preparation

• Vimeo90K: Training / Vid4: Validation
• REDS: Training, Validation
  • download train_sharp data
• after downloading the dataset, use run_downscaling.sh to make lr, slr images
  • sh codes/run_downscaling.sh
• make symbolic link to the datasets.

Downloading Pretrained Models

• You can download pretrained models using the link: [Google Drive]

Usage

• To run EDVR, first install Deformable Convolution. We use mmdetection's dcn implementation. Please first compile it.

  cd ./codes/models/archs/dcn
  python setup.py develop
  

Training

Two ways to train DynaVSR network.

• Distributed training(When using multiple GPUs).

  cd ./codes
  python -m torch.distributed.launch --nproc_per_node=8 --master_port=4321 train_dynavsr.py -opt options/train/[Path to YML file] --launcher pytorch --exp_name [Experiment Name]
  

• Single GPU training.

  cd ./codes
  python train_dynavsr.py -opt options/train/[Path to YML file] --exp_name [Experiment Name]
  

Testing

• We just support single GPU for testing.

  • To use your own test set, change the dataroot in EDVR_Demo.yml file to the folder containing images.

  cd ./codes
  python test_dynavsr.py -opt options/test/[Path to YML file]
  

• Or just use run_visual.sh (Uncomment line that you want to execute).

  sh ./codes/run_visual.sh
  

Results

References

If you like the paper, please cite our paper using:

@inproceedings{lee2021dynavsr,
  title={DynaVSR: Dynamic Adaptive Blind Video Super-Resolution},
  author={Lee, Suyoung and Choi, Myungsub and Lee, Kyoung Mu},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
  pages={2093--2102},
  year={2021}
}

Update Log

• [2020-12-09] First code cleanup, Publish pretrained models
• [2020-12-29] Add arbitrary input data testing script
• [2021-01-05] Add script for testing 4X scale EDVR models
• [2021-01-06] Modify scripts for generating LR, SLR frames

Acknowledgement

The code is built based on

• EDVR-Pytorch

dynavsr's Issues

Consider colab.

I just need to imporve few personal vide files.
Can you please concider to setup your solution in google colab.

Test on my own dataset without GT.

Hi SuYoung,

Can you please comment on how to test DynaVSR on my own dataset without GT?

Thanks:)

Weights of trained models

Hi,

Interesting work! I was wondering whether you could share the weights of trained models for the people who are interested in just testing the model. Furthermore, is using the term sub-low-resolution more appropriate than super-low-resolution since the resolution of image you are referring to is lower than the original low-resolution image?

Training on your own dataset with bilinear downscaling

I already have my own 4x dataset to compare model following the REDS structure (and this is using the basicSR VSR loader, so dataloading shouldn't be an issue unless I need to flatten the folders to pretrain SISR), but for the pretraining, I don't know if MFDN needs to pretrain separately before starting a large training session.
Regarding the TOFLOW and EDVR backed models, which networks should be retrained with the pretraining configs when fitting for a new downscale, patch size(depending on the arch), and subject (low noise, computer generated video)?
Secondly, are there other architectures (like RRDB) in this repo that could be used for VSR training? AFAK dcn/EDVR does not pair well with fp16 and AMP training for consumer cards.
Blind VSR seems far more efficient than SOFVSR(which requires deblurring with HINet for satisfactory results) if it can keep flicker low while also upscaling.

Dataset Preparation

I have noticed that the file 'make_downscaled_images.py' doesn't include the name 'vimeo', can I use this file to prepare dataset from vimeo-90k to train? looking for your reply, thank you!

DynaVSR/codes/make_downscaled_images.py

Line 39 in 39e7abd

prog.add_argument('--dataset_mode', '-m', type=str, default='Vid4+REDS', help='data_mode')

The meta-training code has some mistakes for inner_update.

The variables dictionary optim_params for inner_optimizer contains the parameters of modelcp and est_modelcp. However, these two models are not used during the inner update.
Instead, the models model and 'est_model' are used, whether the parameters contained in optim_params should be related to these two models?

Results in MSU Video Super Resolution Benchmark

Hello,
MSU Video Group has recently launched Video Super Resolution Benchmark and evaluated this algorithm.

DynaVSR-R takes 3rd place by subjective score, 7th place by PSNR, and 3rd by our metric ERQAv1.0. DynaVSR-V takes 13th place by subjective score, 6th place by PSNR, and 10th by our metric ERQAv1.0.
You can see the results here.

If you have any other VSR method you want to see in our benchmark, we kindly invite you to participate.
You can submit it for the benchmark, following the submission steps.