abhiskk / fast-neural-style

pytorch implementation of fast-neural-style

fast-neural-style 🌇 🚀

NOTICE: This codebase is no longer maintained, please use the codebase from pytorch examples repository available at pytorch/examples/fast_neural_style.

This repository contains a pytorch implementation of an algorithm for artistic style transfer. The algorithm can be used to mix the content of an image with the style of another image. For example, here is a photograph of a door arch rendered in the style of a stained glass painting.

The model uses the method described in Perceptual Losses for Real-Time Style Transfer and Super-Resolution along with Instance Normalization. The saved-models for examples shown in the README can be downloaded from here.

DISCLAIMER: This implementation is also a part of the pytorch examples repository. Implementation in this repository uses pretrained Caffe2 VGG whereas the pytorch examples repository implementation uses pretrained Pytorch VGG. The two VGGs have different preprocessings which results in different --content-weight and --style-weight parameters. The styled output images also look slightly different.

Requirements

The program is written in Python, and uses pytorch, scipy. A GPU is not necessary, but can provide a significant speed up especially for training a new model. Regular sized images can be styled on a laptop, desktop using saved models.

Setup the environnment

Run with virtualenv

Create a virtualenv with python3.5 or python3.6. Older versions are not supported due to a lack of compatibilty with pytorch.

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Run with Docker

Build the image:

docker build . -t fast-neural-style

Run the container:

docker run --rm --volume "$(pwd)/:/data" style eval --content-image /data/image.jpg --model /app/saved-models/mosaic.pth --output-image /data/output.jpg --cuda 0

Usage

Stylize image

python neural_style/neural_style.py eval --content-image </path/to/content/image> --model </path/to/saved/model> --output-image </path/to/output/image> --cuda 0

• --content-image: path to content image you want to stylize.
• --model: saved model to be used for stylizing the image (eg: mosaic.pth)
• --output-image: path for saving the output image.
• --content-scale: factor for scaling down the content image if memory is an issue (eg: value of 2 will halve the height and width of content-image)
• --cuda: set it to 1 for running on GPU, 0 for CPU.

Train model

python neural_style/neural_style.py train --dataset </path/to/train-dataset> --style-image </path/to/style/image> --vgg-model-dir </path/to/vgg/folder> --save-model-dir </path/to/save-model/folder> --epochs 2 --cuda 1

There are several command line arguments, the important ones are listed below

• --dataset: path to training dataset, the path should point to a folder containing another folder with all the training images. I used COCO 2014 Training images dataset [80K/13GB] (download).
• --style-image: path to style-image.
• --vgg-model-dir: path to folder where the vgg model will be downloaded.
• --save-model-dir: path to folder where trained model will be saved.
• --cuda: set it to 1 for running on GPU, 0 for CPU.

Refer to neural_style/neural_style.py for other command line arguments.

Models

Models for the examples shown below can be downloaded from here or by running the script download_styling_models.sh.

fast-neural-style's Issues

Python error: <stdin> is a directory, cannot continue

DOVAN$ python neural_style/neural_style.py eval --content-image </Users/DOVAN/Desktop/WechatIMG128.jpeg> --model </Users/DOVAN/deeprely/fast-neural-style/saved-models/starry-night.pth> --output-image </Users/DOVAN/Desktop/test> --cuda 0
Python error: is a directory, cannot continue

Problem with training

Use Reflection Padding

Runtime error while trying out the toy example

I am getting a runtime error w.r.t tensor size while trying out the toy example.

RuntimeError: The expanded size of the tensor (1080) must match the existing size (32) at non-singleton dimension 2. at /opt/conda/conda-bld/pytorch_1502001039157/work/torch/lib/TH/generic/THTensor.c:308

I am using Python 2.7 and used conda install pytorch torchvision -c soumith to install pytorch

Stack trace

File "~/envs/py27/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in __call__ result = self.forward(*input, **kwargs) File "~/fast-neural-style/neural_style/transformer_net.py", line 131, in forward mean = torch.mean(t, 2).unsqueeze(2).expand_as(x) File "~/envs/py27/lib/python2.7/site-packages/torch/autograd/variable.py", line 725, in expand_as return Expand.apply(self, (tensor.size(),)) File "~/envs/py27/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py", line 111, in forward result = i.expand(*new_size)

difference with jcjohnson

Hi, I found there are two differences with jcjohnson's code

1. There is no Tanh after the output conv layer. But in the paper, section 3.1, it said there is a tanh layer after the output conv layer.
2. In the jcjohnson's code, the style image is also scaled to the content image size (256). But here the style image is directly taken into vgg16 without resizing.

Why and what are the influences on the result?

Use instance normalization

RuntimeError: The expanded size of the tensor (700) must match the existing size (32) at non-singleton dimension 2

When I run your code in Pytorch 0.4.0,it will be this error:
RuntimeError: The expanded size of the tensor (700) must match the existing size (32) at non-singleton dimension 2

I think it just because of the version of Pytorch ,Can you fix it,thank you?

download_styling_models.sh error

When I run download_styling_models.sh, it gets error: failed: Connection refused. I couldn't download the pretrained model from the website. Any ideas? Thanks!

Query Regarding Transforms used for input image and style image

What is the purpose of this line in your transforms block for the input :

transforms.Lambda(lambda x: x.mul(255))

Aren't the input and style images already in the range 0-255? Wouldn't this cause an overflow?

About adding Tanh() at the end of the model

Hi, when adding Tanh() at the end of the transformer model, I got error during the training.
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
Is there a way to fix this problem? Thanks!

In the default configuration don't resize the styling-image when training the network

stylize using GPU

Takes about 9s on an image. Looks like it's not using the GPU

The scale of content and style loss just changed from 1e5 (content), 1e3 (style) to 1e2(content), (1e-2) style.

The scale changed after updating to the latest PyTorch release. This involved changing code for loading the Vgg model.

Look into the Vgg code maybe use the Vgg model from torchvision.models.

download_styling_models.sh error

When I run download_styling_models.sh, it gets error：
--2018-03-16 09:26:26-- (try: 2) https://www.dropbox.com/s/gtwnyp9n49lqs7t/saved-models.zip?dl=1
Connecting to www.dropbox.com (www.dropbox.com)|162.125.82.1|:443... failed: Connection timed out.
Connecting to www.dropbox.com (www.dropbox.com)|2620:100:6032:1::a27d:5201|:443... failed: Cannot assign requested address.
Retrying.

Is there any other way to download it?
Thanks！

Could you upload vgg16.weight file?

Hi,

I'm trying to train new styles images on Windows (anaconda, pytorch 0.3).
Using vgg.t7 file gives me the following error messages:

... ...
File "c:\Anaconda3\envs\py36torch\lib\site-packages\torch\utils\serialization\read_lua_file.py", line 572, in read_table
v = self.read()
File "c:\Anaconda3\envs\py36torch\lib\site-packages\torch\utils\serialization\read_lua_file.py", line 598, in read
"corrupted.".format(typeidx))
torch.utils.serialization.read_lua_file.T7ReaderException: unknown type id 1056626884. The file may be corrupted.

I assume this is due to linux/windows compatibility and would like to try with vgg16.weight file.

Thank you for your code.

Add code for stylizing images, use model.eval() when you are stylizing.

Time for training a new style

Hi,
I am using p2.xlarge in AWS (NVIDIA K80 GPU, 61GB RAM , 128GB Space). To check it out I was using COCO's val2017(5K images). It downloaded the vgg-model file and it has been blank for a very long while.
0.How much time does it take to get trained for 5000 images?
1.Also when I am trying to kill the python process with "kill -9 PID", it is not getting killed, is it because of the multi threading ? How do I end the training process in between ?

--> I am able to run 'eval' option using cuda 1 and it takes about 1-2 seconds to generate output images. So I am assuming all the package installations are fine on my side.

Please help me resolve the issues.

Save CPU version of the trained model

How to set output_image_size when Eval

Hi
1、Can you explain what's different of the image_size of "content and style" when Train and Eval?
2、How to set output_image_size when Eval? (if your default setting is 1080,can you show me where you set in your code?)
3、Really thanks for your work

content loss calculation is wrong?

Hi, I found the vgg16 loss network returns [relu1_2, relu2_2, relu3_3, relu4_3], and in your neural_style.py,

y = transformer(x)
features_y = vgg(y)
f_xc_c = Variable(features_xc[1].data, requires_grad=False)
content_loss = args.content_weight * mse_loss(features_y[1], f_xc_c)

the features_y is a list of loss corresponding to [relu1_2, relu2_2, relu3_3, relu4_3], the index for relu1_2, relu2_2, relu3_3 and relu4_3 is 0, 1, 2, and 3, right? In the paper, it used relu3_3 as content loss, so, here should it be the following?
f_xc_c = Variable(features_xc[2].data, requires_grad=False) content_loss = args.content_weight * mse_loss(features_y[2], f_xc_c)
Thanks.

some question in utils.py

def tensor_save_rgbimage(tensor, filename, cuda=False):
    if cuda:
        img = tensor.clone().cpu().clamp(0, 255).numpy()
    else:
        img = tensor.clone().clamp(0, 255).numpy()
    img = img.transpose(1, 2, 0).astype('uint8')
    img = Image.fromarray(img)
    img.save(filename)


def tensor_save_bgrimage(tensor, filename, cuda=False):
    (b, g, r) = torch.chunk(tensor, 3)
    tensor = torch.cat((r, g, b))
    tensor_save_rgbimage(tensor, filename, cuda)

it seems like if you use function tensor_save_bgrimage to save a tensor, it will first transform from BGR to RGB and then transform from RGB to GBR in the function tensor_save_rgbimage?

Use Upsampling instead of ConvTranspose2d

Use median filter

train doesn't run: ValueError: not enough values to unpack (expected 3, got 2)

Hello. train doesn't run:

danusya@werewolf:~/fast-neural-style$ python3 neural_style/neural_style.py train --dataset ~/dataset/ --style-image ~/Caleido/Color_Kaleidoscope_2017-6-24_15141.png  --vgg-model-dir caleido_vgg --save-model-dir caleido_model --epochs 2 --cuda 0
/usr/local/lib/python3.5/dist-packages/torchvision/transforms/transforms.py:156: UserWarning: The use of the transforms.Scale transform is deprecated, please use transforms.Resize instead.
  "please use transforms.Resize instead.")
Traceback (most recent call last):
  File "neural_style/neural_style.py", line 210, in <module>
    main()
  File "neural_style/neural_style.py", line 204, in main
    train(args)
  File "neural_style/neural_style.py", line 50, in train
    style = utils.preprocess_batch(style)
  File "/home/danusya/fast-neural-style/neural_style/utils.py", line 59, in preprocess_batch
    (r, g, b) = torch.chunk(batch, 3)
ValueError: not enough values to unpack (expected 3, got 2)

Check usage of 'subtract_imagenet_mean', with new version of pytorch inplace subtraction might be different.

trained model not working

epoch_2_Sat_Oct_21_17:31:07_2017_1.0_5.0.model.tar.gz
i have trained a model, program has finished successfully, yet evaluating the model seems to produce a blank screen.
i have attached the model.
https://www.dropbox.com/s/r9fxsy129he7rd1/epoch_2_Sat_Oct_21_17%3A31%3A07_2017_1.0_5.0.model.tar.gz?dl=0
please help ):