hughplay / dfnet

:art: Deep Fusion Network for Image Completion - ACMMM 2019

Deep Fusion Network for Image completion

Official repository for "Deep Fusion Network for Image completion".

Figure: Results from DFNet. Fusion Result = (1 - Alpha) * Input + Alpha * Raw. Both Raw and Alpha are model outputs.

Deep Fusion Network for Image Completion
Xin Hong, Pengfei Xiong, Renhe Ji, Haoqiang Fan
Published on Proceedings of the 27th ACM International Conference on Multimedia (ACMMM 2019)

Description

Deep image completion usually fails to harmonically blend the restored image into existing content, especially in the boundary area. And it often fails to complete complex structures.

We first introduce Fusion Block for generating a flexible alpha composition map to combine known and unknown regions. It builds a bridge for structural and texture information, so that information in known region can be naturally propagated into completion area. With this technology, the completion results will have smooth transition near the boundary of completion area. Furthermore, the architecture of fusion block enable us to apply multi-scale constraints. Multi-scale constrains improves the performance of DFNet a lot on structure consistency.

Moreover, it is easy to apply this fusion block and multi-scale constrains to other existing deep image completion models. A fusion block feed with feature maps and input image, will give you a completion result in the same resolution as given feature maps.

If you find this code useful, please consider to star this repo and cite us:

@inproceedings{hongDeepFusionNetwork2019,
  title = {Deep {{Fusion Network}} for {{Image Completion}}},
  booktitle = {Proceedings of the 27th {{ACM International Conference}} on {{Multimedia}}},
  author = {Hong, Xin and Xiong, Pengfei and Ji, Renhe and Fan, Haoqiang},
  year = {2019},
  series = {{{MM}} '19},
  pages = {2033--2042},
  keywords = {alpha composition,deep fusion network,fusion block,image completion,inpainting}
}

Prerequisites

• Python 3
• PyTorch 1.0
• OpenCV

Testing

We provide an interactive Colab demo for trying DFNet. You can also test our model with the following steps.

Clone this repo:

git clone https://github.com/hughplay/DFNet.git
cd DFNet

Download pre-trained model from Google Drive and put them into model.

Testing with Places2 model

There are already some sample images in the samples/places2 folder.

python test.py --model model/model_places2.pth --img samples/places2/img --mask samples/places2/mask --output output/places2 --merge

Testing with CelebA model

There are already some sample images in the samples/celeba folder.

python test.py --model model/model_celeba.pth --img samples/celeba/img --mask samples/celeba/mask --output output/celeba --merge

Training

Please refer to: https://github.com/deepcodebase/inpaint. It is building in progress but looks good so far.

License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

dfnet's Issues

Resulting Images Size

Is it possible to set the final inpainted image size same as the input images?

I tried the same and got this error:

Using cpu.
Model model/model_places2.pth loaded.

Inpainting...

Input size: (500, 333)
Traceback (most recent call last):
File "test.py", line 263, in
tester.inpaint(args.output, args.img, args.mask, merge_result=args.merge)
File "test.py", line 221, in inpaint
self.process_batch(batch, output)
File "test.py", line 172, in process_batch
result, alpha, raw = self.model(imgs_miss, masks)
File "/home/sadbhawna/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/sadbhawna/torch/DFNet/model.py", line 260, in forward
out = decode(out, out_en[-i-2])
File "/home/sadbhawna/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/sadbhawna/torch/DFNet/model.py", line 147, in forward
out = torch.cat([out, concat], dim=1)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 2 and 3 in dimension 3 at /opt/conda/conda-bld/pytorch_1574150980135/work/aten/src/TH/generic/THTensor.cpp:612

Training code

Please provide the training code.

Edge is not painting

I have did some tests on Pre-trained model but the missing edges weren't fill. it's seem the edges is not be connected.

ONNX conversion

Hi ,is it possible to convert DFNet pytorch model to ONNX format?

something about training

Hi Mr Blue,
here I have some questions about training, Could you please give me some suggestions?

1. which layers are the perceptual and style loss used (iin my opinion, the results of resolution with 256, 128, 64 if I trained celeba, is that right? the index in loss.py is [0,1,2])
2. the input of vgg is the results of model's output, is that right?
3. in your loss.py the vgg feature extractor don't sub(mean) and div(std), why?
4. the reconstruction loss don't consider the hole and valid region (like Nvidia partialconv)

Add a way to extract an `ONNX` file

First of all, thanks a lot for this amazing project and your hard work!

In https://github.com/burn-rs/burn we are trying to load different ONNX models, convert them into burn models, and then use them on different backends. It would be fantastic if we could try your model too. Can you add a way to extract the ONNX model? Using PyTorch is quite feasible.

Thanks a lot in advance!

Training code

Training code is necessary to understand what loss functions were used, what was the model architecture etc. It would be a nice idea to add training code too.

Can I just use DataLoader in Pytorch?

Hi! Thanks for your amazing work. I'm currently adapting your code to train on my own dataset, with slightly different data format. So I'm thinking about tweaking your data loading module a bit to suit my needs. However, your data loading module seems pretty convoluted, with multithreading and buffers, that I found hard to understand.

I'm wondering, is your code functionally equivalent as torch.data.utils.DataLoader? Can I just implement my own Dataset object and use DataLoader for multi-thread reading? Will it be significantly slower than your implementation? Thanks!

Output Result ?

Hello, thank you for your amazing implementation,
I tried running the inference on custom images, the execution completed but the resulted images are not saved in the directory,
it'll be much appreciated if you would look into this issue
Thank you

Are these two assignments reversed in loss.py

text_r = [targets[i] for i in self.l_text]
text_t = [results[i] for i in self.l_text]
Are these two assignments reversed in loss.py

lambda parameters for training

Hi,

I am trying to implement this model, but cannot find lambda parameters setting from your great paper. Like for P=1, Q=1 how to weighted sum them in loss?

thanks