GithubHelp home page GithubHelp logo

seranus / arxiv2020-rife Goto Github PK

View Code? Open in Web Editor NEW

This project forked from hzwer/eccv2022-rife

1.0 1.0 2.0 7.73 MB

RIFE: Real Time Video Frame Rate Increasement

Home Page: https://rife-vfi.github.io/

License: MIT License

Python 95.10% Jupyter Notebook 4.90%

arxiv2020-rife's Introduction

RIFE Video Frame Interpolation

11.22 News: We notice a new windows app is trying to integrate RIFE, we hope everyone to try and help them improve. You can download Flowframes for free.

We find a tutorial of RIFE on Youtube.

11.20 News: I optimize the parallel processing, get 60% speedup!

Date of recent model update: 2020.11.19, v1.2

Our model is currently not very suitable for 2d animation.

You can easily use colaboratory to have a try and generate the our youtube demo.

Our model can run 30+FPS for 2X 720p interpolation on a 2080Ti GPU. Currently our method supports 2X,4X,8X interpolation for 1080p video, and multi-frame interpolation between a pair of images. Everyone is welcome to use our alpha version and make suggestions!

16X interpolation results from two input images:

Demo Demo

Abstract

We propose RIFE, a Real-time Intermediate Flow Estimation algorithm for Video Frame Interpolation (VFI). Most existing methods first estimate the bi-directional optical flows and then linearly combine them to approximate intermediate flows, leading to artifacts on motion boundaries. RIFE uses a neural network named IFNet that can directly estimate the intermediate flows from images. With the more precise flows and our simplified fusion process, RIFE can improve interpolation quality and have much better speed. Based on our proposed leakage distillation loss, RIFE can be trained in an end-to-end fashion. Experiments demonstrate that our method is significantly faster than existing VFI methods and can achieve state-of-the-art performance on public benchmarks.

Usage

Installation

git clone [email protected]:hzwer/arXiv2020-RIFE.git
cd arXiv2020-RIFE

pip install -r requirements.txt
  • Download the pretrained models from here. We are optimizing the visual effects and will support animation in the future.

(我们也提供了百度网盘链接:https://pan.baidu.com/s/17mK2oTZUCMtMgmAdoifLGA 密码:h0cl,把压缩包解开后放在 train_log/*.pkl)

  • Unzip and move the pretrained parameters to train_log/*.pkl

The models under different setting is coming soon.

Run

Video Frame Interpolation

You can use our demo video or use your own video to process.

$ python3 inference_video.py --exp=1 --video=video.mp4 

(generate video_2X_xxfps.mp4)

$ python3 inference_video.py --exp=2 --video=video.mp4

(for 4X interpolation)

$ python3 inference_video.py --exp=2 --video=video.mp4 --fps=60

(add slomo effect, the audio will be removed)

$ python3 inference_video.py --video=video.mp4 --montage --png

(if you want to montage the origin video, and save the png format output)

$ python3 inference_video_parallel.py --exp=2 --video=video.mp4

(Try our parallel process, may be useful on your device.)

The warning info, 'Warning: Your video has *** static frames, it may change the duration of the generated video.' means that your video has changed the frame rate by adding static frames, it is common if you have processed 25FPS video to 30FPS.

Image Interpolation

$ python3 inference_img.py --img img0.png img1.png --exp=4

(2^4=16X interpolation results) After that, you can use pngs to generate mp4:

$ ffmpeg -r 10 -f image2 -i output/img%d.png -s 448x256 -c:v libx264 -pix_fmt yuv420p output/slomo.mp4 -q:v 0 -q:a 0

You can also use pngs to generate gif:

$ ffmpeg -r 10 -f image2 -i output/img%d.png -s 448x256 -vf "split[s0][s1];[s0]palettegen=stats_mode=single[p];[s1][p]paletteuse=new=1" output/slomo.gif

Evaluation

First you should download RIFE model reported by our paper.

We will release our training and benchmark validation code soon.

Vimeo90K: Download Vimeo90K dataset at ./vimeo_interp_test

MiddleBury: Download MiddleBury OTHER dataset at ./other-data and ./other-gt-interp

$ python3 benchmark/Vimeo90K.py
(Final result: "Avg PSNR: 35.695 SSIM: 0.9788")
$ python3 benchmark/MiddelBury_Other.py
(Final result: "2.058")

Citation

img

@article{huang2020rife,
  title={RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation},
  author={Huang, Zhewei and Zhang, Tianyuan and Heng, Wen and Shi, Boxin and Zhou, Shuchang},
  journal={arXiv preprint arXiv:2011.06294},
  year={2020}
}

Reference

Optical Flow: ARFlow pytorch-liteflownet RAFT pytorch-PWCNet

Video Interpolation: DAIN CAIN AdaCoF-pytorch

arxiv2020-rife's People

Contributors

a1600012888 avatar dynmi avatar eonzenex avatar hzwer avatar sloganking avatar stonecypher avatar

Stargazers

 avatar

Watchers

 avatar

Forkers

bertwang amesianx

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.