Fast-Neural-Style-Transfer

Generative Neural Methods Based On Model Iteration

Description

Using a Deep Residual Convolutional Neural Network as an Image Transformation Network (ITN). We train the ITN to transform input images into output images. We use a VGG19 which is pre-trained on ImageNet dataset as Loss Network to define two Perceptual Losses (Feature Reconstruction Loss & Style Reconstruction Loss) that measure perceptual differences in content and style between images.
For a specific style, we train one ITN using MS-COCO dataset (about 12.6GB). After trained, we can use the ITN to transfer specific style to any image just performing one feed-forward computation.

This code is based on Johnson et al. Perceptual Losses for Real-Time Style Transfer and Super-Resolution [2016.03] and Ulyanov et al. Instance Normalization: The Missing Ingredient for Fast Stylization [2016.09].

System overview. Picture comes from Johnson et al. original paper. In my implementation, I replace the VGG-16 to a pre-trained VGG-19. I use 'relu4_2' layer to compute Feature Reconstruction Loss and use {'relu1_1', 'relu2_1', 'relu3_1', 'relu4_1', 'relu5_1'} layers to compute Style Reconstruction Loss.

Results

style	output (generated image)

Manual

The main file main.py is a demo, which has already contained training procedure and inferring procedure (inferring means generating stylized images).
You can switch these two procedures by changing the flag IS_TRAINING.
By default,
(1) The content images lie in the folder "./images/content/"
(2) The style images lie in the folder "./images/style/"
(3) The weights file of the pre-trained VGG-19 lies in the current working directory. (See Prerequisites below)
(4) The MS-COCO images dataset for training lies in the folder "./MS_COCO/" (See Prerequisites below)
(5) The checkpoint files of trained models lie in the folder "./models/" (You should create this folder manually before training)
(6) After inferring procedure, the stylized images will be generated and put in the folder "./outputs/"
For training, you should make sure (2), (3), (4) and (5) are prepared correctly.
For inferring, you should make sure (1) and (5) are prepared correctly.
Of course, you can organize all the files and folders as you want, and what you need to do is just modifying the main.py file.

Prerequisites

Pre-trained VGG19 network (MD5 8ee3263992981a1d26e73b3ca028a123)
I have provided a convertor in the tool folder. It can convert the matlab file into a npz file which is much smaller and easier to process via NumPy.
Or simply download my pre-processed Pre-trained VGG19 network npz format (MD5 c7ddd13b12e40033b5031ff43e467065) The npz format is about 80MB while the mat format is about 550MB.
Chinese user can also use BaiduYun to download from here.
Microsoft COCO dataset

Trained Models

I have trained models over 10 styles: wave, udnie, escher_sphere, flower, scream, denoised_starry, starry_bright, rain_princess, woman_matisse, mosaic. (You can click the "models" to download)
Chinese user can also use BaiduYun to download from here.

My Running Environment

Hardware

CPU: Intel® Core™ i9-7900X (3.30GHz x 10 cores, 20 threads)
GPU: NVIDIA® Titan Xp (Architecture: Pascal, Frame buffer: 12GB)
Memory: 32GB DDR4

Operating System

ubuntu 16.04.03 LTS

Software

Python 3.6.2
NumPy 1.13.1
TensorFlow 1.3.0
SciPy 0.19.1
CUDA 8.0.61
cuDNN 6.0.21

References

This project borrowed some ideas and paradigms from Logan Engstrom's Fast Style Transfer and Zhiyuan He's Fast Neural Style Tensorflow
Parts of README formatting were copied from Justin Johnson's Fast Neural Style and Zhiyuan He's Fast Neural Style Tensorflow

Citation

  @misc{ye2017fastneuralstyletransfer,
    author = {Wengao Ye},
    title = {Fast Neural Style Transfer},
    year = {2017},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/elleryqueenhomels/fast-neural-style-transfer}}
  }

lycokie / fast_neural_style_transfer Goto Github PK