luanfujun / deep-painterly-harmonization

Code and data for paper "Deep Painterly Harmonization": https://arxiv.org/abs/1804.03189

deep-painterly-harmonization

Code and data for paper "Deep Painterly Harmonization"

Disclaimer

This software is published for academic and non-commercial use only.

Setup

This code is based on torch. It has been tested on Ubuntu 16.04 LTS.

Dependencies:

• Torch (with loadcaffe)
• Matlab or Octave

CUDA backend:

• CUDA
• cudnn

Download VGG-19:

sh models/download_models.sh

Compile cuda_utils.cu (Adjust PREFIX and NVCC_PREFIX in makefile for your machine):

make clean && make

Usage

To generate all results (in data/) using the provided scripts, simply run

python gen_all.py

in Python and then

run('filt_cnn_artifact.m')

in Matlab or Octave. The final output will be in results/.

Note that in the paper we trained a CNN on a dataset of 80,000 paintings collected from wikiart.org, which estimates the stylization level of a given painting and adjust weights accordingly. We will release the pre-trained model in the next update. Users will need to set those weights manually if running on their new paintings for now.

Removed a few images due to copyright issue. Full set here for testing use only.

Examples

Here are some results from our algorithm (from left to right are original painting, naive composite and our output):

Acknowledgement

• Our torch implementation is based on Justin Johnson's code;
• Histogram loss is inspired by Risser et al.

Citation

If you find this work useful for your research, please cite:

@article{luan2018deep,
  title={Deep Painterly Harmonization},
  author={Luan, Fujun and Paris, Sylvain and Shechtman, Eli and Bala, Kavita},
  journal={arXiv preprint arXiv:1804.03189},
  year={2018}
}

Contact

Feel free to contact me if there is any question (Fujun Luan [email protected]).

deep-painterly-harmonization's Issues

Nice code

I am wondering how you implement your code 'cuda_utils.cu'.
I am suffering from how to write efficient cuda code to make my custom operation work.
Could you give me some advices ?
BTW, I will check your code and may make some PRs to refine your code.

handling out of memory

Thanks very much for this inspiring work. I'm trying to run this on an NVIDIA Quadro P4000 and am getting:

...
Setting up style layer  	12	:	relu3_1
THCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-7191/cutorch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory
/home/localadmin/torch/install/bin/luajit: .../localadmin/torch/install/share/lua/5.1/nn/Container.lua:67:
In 11 module of nn.Sequential:
cuda runtime error (2) : out of memory at /tmp/luarocks_cutorch-scm-1-7191/cutorch/lib/THC/generic/THCStorage.cu:66
stack traceback:
	[C]: at 0x7f1471b9dcc0
	[C]: in function 'cmul'
	neural_paint.lua:576: in function <neural_paint.lua:575>
	[C]: in function 'xpcall'
	.../localadmin/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'
	...localadmin/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
	neural_paint.lua:237: in function 'main'
	neural_paint.lua:738: in main chunk
	[C]: in function 'dofile'
	...dmin/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
	[C]: at 0x00405d50

Do you have any pointers on how one might go about resolving an OOM result like this?

"attempt to index a nil value" while trying to reproduce the results

I get

luajit: neural_paint.lua:140: attempt to index a nil value
stack traceback:
        neural_paint.lua:140: in function 'main'
        neural_paint.lua:739: in main chunk
        [C]: in function 'dofile'
        /usr/bin/th:150: in main chunk
        [C]: at 0x00404680

while trying to reproduce the results while folllowing the directions in the README.

Full logs:

Successfully loaded models/VGG_ILSVRC_19_layers.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000
Running optimization with L-BFGS
   creating recyclable direction/step/history buffers
   function value changing less than tolX
Gpu, idx      =         0       1
idx=0, classified label=Art Nouveau (Modern), weight=1.203930
Successfully loaded models/VGG_ILSVRC_19_layers.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000
Working on patchmatch layer     4       :       relu4_1
luajit: neural_paint.lua:140: attempt to index a nil value
stack traceback:
        neural_paint.lua:140: in function 'main'
        neural_paint.lua:739: in main chunk
        [C]: in function 'dofile'
        /usr/bin/th:150: in main chunk
        [C]: at 0x00404680

extend the code to support CPU

Hi,

Do you think, it'd be hard to reproduce the code in CPU? As far as I know, the dense correspondence search and spatial consistency resampling are implemented in CUDA? And if it is possible, would it be time too much time consuming? I'll appreciate any help.

Best

Conflicts between paper and the code

Hi I'm reading through your neural_paint.lua and I'm noticing a lot of conflicts
At https://github.com/luanfujun/deep-painterly-harmonization/blob/master/neural_paint.lua#L151
You use conv5_1 as your reference layer, yet in the paper, you say that conv4_1 was the optimum

At line 155, you use the resized style image as your reference guide, but in the paper, you again say that conv4_1 is the reference layer

At line 168, it appears you are storing the un-matched style features and not the matched features and are completely ignoring the matched and then consistency refined features

Can you explain these conflicts?

Thanks

any luck getting this going with fastAI on colab?

Seems like no one has been able to do this? I have been fiddling for a few hours, figured I was ask here while still working. thanks.

这篇论文是在哪个大会上获得最佳论文的？

Run 'make clean && make ' error

Platform: Mac OS 10.13.5 (17F77)

Here is the error message below.

find . -type f | xargs -n 5 touch
rm -f libcuda_utils.so
/usr/local/cuda/bin/nvcc -arch sm_35 -O3 -DNDEBUG -Xcompiler -fopenmp --compiler-options '-fPIC' -o libcuda_utils.so --shared cuda_utils.cu -I/Users/xxx/torch/install/include/THC -I/Users/xxx/torch/install/include/TH -I/Users/xxx/torch/install/include -L/Users/xxx/torch/install/lib -Xlinker -rpath,/Users/xxx/torch/install/lib -lluaT -lTHC -lTH -lpng -lluajit -lgomp
nvcc fatal : The version ('9.1') of the host compiler ('Apple clang') is not supported
make: *** [libcuda_utils.so] Error 1

clang --version

Apple LLVM version 9.1.0 (clang-902.0.39.2)
Target: x86_64-apple-darwin17.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

no matlab

Hi, we can not run matlab, does your program can run without matlab?

~

cuda_utils.cu:8:17: fatal error: THC.h: 没有那个文件或目录
compilation terminated.
makefile:11: recipe for target 'libcuda_utils.so' failed
make: *** [libcuda_utils.so] Error 1

/usr/bin/ld: cannot find -lluajit

torch & lua 5.1 are installed

But still not working

May I get some advice

having trouble for 3 days...

@luanfujun

gradient computation of StyleLoss in neural_gram.lua

@luanfujun, when I read the code function StyleLoss:updateGradInput in neural_gram.lua, I find maybe there is something wrong about the gradient computation, after

self.gradInput:mul(self.strength)

we should firstly do

self.gradInput:cmul(msk)

and then do

self.gradInput:add(gradOutput)

lua.h: No such file or directory

Trying to run 'make' on a SageMaker hosted Jupyter Notebook. Getting this error:

sh-4.2$ make
/usr/local/cuda-8.0/bin/nvcc -arch sm_35 -O3 -DNDEBUG -Xcompiler -fopenmp --compiler-options '-fPIC' -o libcuda_utils.so --shared cuda_utils.cu -I/home/ubuntu/torch/install/include/THC -I/home/ubuntu/torch/install/include/TH -I/home/ubuntu/torch/install/include -L/home/ubuntu/torch/install/lib -Xlinker -rpath,/home/ubuntu/torch/install/lib -lluaT -lTHC -lTH -lpng -lluajit -lgomp
cuda_utils.cu:2:18: fatal error: lua.h: No such file or directory
#include "lua.h"
^
compilation terminated.
make: *** [libcuda_utils.so] Error 1
sh-4.2$

Full wikiart_output.txt

When the model was trained on all the wikiart images, was a file like the wikiart_output.txt generated with entries for every image in the training set? Would it be possible to get a copy of the file with all entries?

Thanks for your time and effort, really appreciate what has been done.

Will it have a pytorch version?

Will it have a pytorch version? I would very appreciate it.

Running filt_cnn_artifact.m in Octave

Will the filt_cnn_artifact.m code run in Octave as developed? I am having issues with the nnmex and votemex code calls not returning values. Thanks.

Does it handle large images?

I tried training it with a 1440 image and it throws an out of memory error. I did use --image_size 1440

a working 2022 colab!

You guys asked for it! building on @marcduda 's efforts, here is a 2022 fully working colab with minimal GUI. it can also calculate the mask for you - just upload the naive and target images and play with the style weight (as the wikiart predictor was never released afaiu).
https://colab.research.google.com/gist/eyaler/5303782669fb43510d398bd346c6e3e6/deep-painterly-harmonization.ipynb

Would you please publish a detailed usage (including train my own mask,generate result)in README.md?

Would you please publish a detailed usage (including train my own mask,generate result)in README.md? I'm just a senior high student ,and it's difficult for me to understand all this.I just wonna know how to use it :-P Thanks.

Setup process

Hi, I'm having problems installing the dependencies, could you add a list of the versions you used? or upload an image from ubuntu 16.04 with dependencies added to amazon?

CUDA

I tried to write the part of matlab dealing with mask in gen_all.py, but I got an error:

/home/wyy/torch/install/bin/luajit: neural_gram.lua:54: cuda runtime error (10) : invalid device ordinal at /home/wyy/torch/extra/cutorch/init.c:734
stack traceback:
[C]: in function 'setDevice'
neural_gram.lua:54: in function 'main'
neural_gram.lua:450: in main chunk
[C]: in function 'dofile'
.../wyy/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50

can you give me some advice? Thank you very much!!!!!

Figure out the right weight for a new painting

Anybody know a way to figure out the right weight for a new painting? On the readme it seems a bit like a manual trial and error thing for now….
Note that in the paper we trained a CNN on a dataset of 80,000 paintings collected from wikiart.org, which estimates the stylization level of a given painting and adjust weights accordingly. We will release the pre-trained model in the next update. Users will need to set those weights manually if running on their new paintings for now.

require 'libcuda_utils' Segmentation fault (core dumped)

Hi, this is a great work.

I am trying to run it on CUDA 9.1, but failed. When I load the libcuda_utils.so library in torch, I get the "Segmentation fault " error. Then, I fall back to CUDA 8.0 and recompile. Still get the same problem. I run other torch project like style transfer. It runs good.

This is surely some version conflicts when compiling libcuda_utils.so. But I am not sure which ones. Could you give some hints?

make
/usr/local/cuda-9.1/bin/nvcc -arch sm_35 -O3 -DNDEBUG -Xcompiler -fopenmp --compiler-options '-fPIC' -o libcuda_utils.so --shared cuda_utils.cu -I/home/chris/torch/install/include/THC -I/home/chris/torch/install/include/TH -I/home/chris/torch/install/include -L/home/chris/torch/install/lib -Xlinker -rpath,/home/chris/torch/install/lib -lluaT -lTHC -lTH -lpng -lluajit -lgomp

Is it possible to modify this model to achieve real-time results.

Marvellous work! The result is real a magic.

However, one may spend several minutes using this model to generate an output. I think the reason is that this model is based on online image optimization. Meanwhile, I find that there are some faster neural style transfer models based on offline optimization.

I am confused that if there is possibly a way to modify this model to be offline. Maybe there is no painterly harmonization projects like that, but I just need some advice.

Thanks!

error while run('filt_filt_cnn_artifact.m')

I ran gen_all.py and it worked well. I got a lot of pictures ,in which each type of image has ten images.
Howerer,when I used run('filt_filt_cnn_artifact.m') in octave, it returned error like this:

invalid use of script /home/jeffrey/cyf/deep-painterly-harmonization-master/3rdparty/patchmatch-2.0/nnmex.m in index expression error: called from
filt_cnn_artifact at line 50 column 9
run at line 86 column 5

Is someone has met this problem and solve it? Please help me ,thank you very much!

Is matlab necessary?

As title.
Can I run your demo in docker without Matlab or Octave.
For me, installing matlab on the server is very difficult, especially in the company.
And I recommend labeling the minimum version of the software/library on which the demo depends.

Please forgive me for my poor English.

Error during compilation of cuda_utils.cu

What I got from the command make clean && make is as follows,

In file included from /usr/local/cuda-9.1/bin/../targets/x86_64-linux/include/host_config.h:50:0,
from /usr/local/cuda-9.1/bin/../targets/x86_64-linux/include/cuda_runtime.h:78,
from :0:
/usr/local/cuda-9.1/bin/../targets/x86_64-linux/include/crt/host_config.h:121:2: error: #error -- unsupported GNU version! gcc versions later than 6 are not supported!
#error -- unsupported GNU version! gcc versions later than 6 are not supported!
^~~~~
makefile:10: recipe for target 'libcuda_utils.so' failed
make: *** [libcuda_utils.so] Error 1

This is my makefile content,

PREFIX=/home/whu/apps/torch/install
NVCC_PREFIX=/usr/local/cuda-9.1/bin
CFLAGS=-I$(PREFIX)/include/THC -I$(PREFIX)/include/TH -I$(PREFIX)/include
LDFLAGS_NVCC=-L$(PREFIX)/lib -Xlinker -rpath,$(PREFIX)/lib -lluaT -lTHC -lTH -lpng -lluajit -lgomp
all: libcuda_utils.so
libcuda_utils.so: cuda_utils.cu
$(NVCC_PREFIX)/nvcc -arch sm_35 -O3 -DNDEBUG -Xcompiler -fopenmp --compiler-options '-fPIC' -o libcuda_utils.so --shared cuda_utils.cu $(CFLAGS) $(LDFLAGS_NVCC)
clean:
find . -type f | xargs -n 5 touch
rm -f libcuda_utils.so

Get error "/torch/extra/cutorch/lib/THC/THCGeneral.c line=70 error=30 : unknown error"

I got error when I run "python gen_all.py". Could you give any help to this problem?

/usr/bin/ld: cannot find -lpng

satoshi@nakamoto:~/deep-painterly-harmonization$ make clean && make
find . -type f | xargs -n 5 touch
rm -f libcuda_utils.so  
/usr/local/cuda-10.0/bin/nvcc -arch sm_35 -O3 -DNDEBUG -Xcompiler -fopenmp --compiler-options '-fPIC' -o libcuda_utils.so --shared cuda_utils.cu -I/home/satoshi/torch/install/include/THC -I/home/satoshi/torch/install/include/TH -I/home/satoshi/torch/install/include -L/home/satoshi/torch/install/lib -Xlinker -rpath,/home/satoshi/torch/install/lib -lluaT -lTHC -lTH -lpng -lluajit -lgomp
In file included from cuda_utils.cu:15:0:
/usr/local/cuda-10.0/bin/..//include/math_functions.h:54:2: warning: #warning "math_functions.h is an internal header file and must not be used directly.  This file will be removed in a future CUDA release.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "math_functions.h is an internal header file and must not be used directly.  This file will be removed in a future CUDA release.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
In file included from cuda_utils.cu:15:0:
/usr/local/cuda-10.0/bin/..//include/math_functions.h:54:2: warning: #warning "math_functions.h is an internal header file and must not be used directly.  This file will be removed in a future CUDA release.  Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
 #warning "math_functions.h is an internal header file and must not be used directly.  This file will be removed in a future CUDA release.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
  ^~~~~~~
/usr/bin/ld: cannot find -lpng
collect2: error: ld returned 1 exit status
make: *** [makefile:10: libcuda_utils.so] Error 1

followed instructions as well as I could. Running Deepin 15.8 which is very ubuntulike and debian based. using CUDA 10 and torch7. no idea what is this lpng that is missing, any tips?

make on CPU

Is there anyway we can make using CPU only? Basically not use CUDA (have no GPU), and still able to use the pretrained model to get some results on some data provided.

Can this transfer to Android？

I want to build it on Android environment，but I have no idea about how to do with it.

makefile:10: recipe for target 'libcuda_utils.so' failed

I was able to correct this through a combination of reinstalling ubuntu 18.04, torch, gcc, and multiple versions of cuda

Requesting to explain the process to run the project in detail

Can you please provide the steps to be followed to setup and run the project in detail.

.

Can I train my own model? how to generate mask & mask_dilated images?

1.How can I train my model using specific style images? and how to choose weights similar to the weights in 'wikiart_output.txt'?
2.I saw mask & mask_dilated images used during process ,how to generate these?

cuda_utils.cu(462): error: identifier "THCudaIntTensor_zero" is undefined

What is the use of mapping?

Hi, I am new to neural style transfer and confused about mapping in this paper. I've learned that mapping establishes an relationship between content layer and target layer. But I don't know how it could help the reconstruction.

So I turned to read the code.

        local match, correspondence = 
            cuda_utils.patchmatch_r(input, target, params.patchmatch_size, 1)
        match:cmul(mask) --match = elementwise multilication of match and mask
        local target_gram = gram:forward(match):clone()
        target_gram:div(mask:sum()) -- target_gram/mask:sum()

        local norm = params.normalize_gradients
        local loss_module = nn.StyleLoss(params.style_weight, target_gram, norm, mask_image):float():cuda()

In the reconstruction part of neural_gram.lua, target_gram is the formed by the patchmatch's result match.
However, in neural_paint.lua, target_gram seems to come from BP, and BP is just the feature extracted by the pretrained VGG net.

    if i == #layers then -- i = 5, relu5_1
      print("  Initializing NNF in layer ", i, ":", name, " with patch ", params.patchmatch_size) 
      print("  Brute-force patch matching...")
      local init_corr = cuda_utils.patchmatch(N_A, N_BP, params.patchmatch_size) --[x1, y1, x2, y2,...,xn, yn]
      local guide = image.scale(style_image, w, h, 'bilinear'):float():cuda()
      print("  Refining NNF...")
      -- enforce spatial consistency
      corr = cuda_utils.refineNNF(N_A, N_BP, init_corr, guide, tmask, params.refine_size, params.refine_iter)
      mask = cuda_utils.Ring2(N_A, N_BP, corr, params.ring_radius, tmask)
      
      curr_corr = corr 
      curr_mask = mask       
    else -- i = 4, relu4_1
      print("  Upsampling NNF in layer ", i, ":", name)
      curr_corr = cuda_utils.upsample_corr(corr, h, w)
      curr_mask = image.scale(mask:double(), w, h, 'simple'):cudaInt()
    end   
      
    table.insert(match_features, BP)
    table.insert(match_masks, curr_mask)
  end 

Meanwhile, the whole patchmatch part has two results, i.e. curr_corr and curr_mask. But it seems that only 'curr_mask' is saved while the other one is abandoned. In my opinion, curr_mask is just the mask adjusted to the patchmatch and curr_corr is more important.

Could anyone kindly help me and give me some explanations? Thanks a lot.

what is wikiart_output.txt used to?

It seems to define many weight parameters. These weights are used to tune the style weights? How can we decide the value for a new image? Thanks.

Nil Value

Get error
neural_paint.lua:507: attempt to perform arithmetic on local 'sty_lev' (a nil value)
stack traceback:
neural_paint.lua:507: in function 'params_wikiart_genre'
neural_paint.lua:101: in function 'main'
neural_paint.lua:738: in main chunk
[C]: in function 'dofile'
...itai/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50

Under Ubuntu 16.04
any suggestions?

painting estimator

I'm looking forward to pre-trained model of painting estimator, too. Could you release it soon? Thanks very much!!

Help..I have a try of pytorch and meet pixel problem

It's a nice work and I want to use pytorch to re-accomplish it.
But during the first step 'IndependentMapping' I meet a pixel mess problem...

I do clamp and use mask image to do backward but the pixels seem to be out of range.
The loss function I use are content loss,gram loss, and TV loss. I haven't use histogram loss.
The model I use is Vgg19 from pytorch model zoo which the data range is 0-1.I'm sure the image I put into is right format(RGB) and when I tune the style weight or content the result changes a bit. I have no clear idea where the problem is.
Can you help me?Thanks!
貌似我该说中文？

Looking for pre-trained model in the next update

Looking for pre-trained model in the next update, it must be really interesting!!

Logo Design

Hello, I would like to design a logo for your software. I can make some ideas if you are interested? Do you have something on your mind for your logo?
Greetings
Nuno

Is this possible to run on mac os?

Hi!
First of all, that's some amazing work here. I wanted to give it a try on Mac OS, but I get errors for some reasons after make clean && make.

I updated PREFIX and NVCC_PREFIX with the right paths, but I guess lua might not be correctly installed. If you have any ideas on how to run this on a Mac environment, I'd be really interested about trying this out! Thanks a lot!

Windows binary

This is such a cool project. If there is a way to make a windows binary that would be awesome and I would be grateful.

Can you add license to this repo?

It will be great if you can add a license to this repo. Thanks :)