shi-labs / fcf-inpainting Goto Github PK

Dear reasercher, please also consider checking our newly introduced face inpainting method to address the symmetry problems of general inpainting mehthods by using swin transformer and semantic aware discriminators.
Our proposed method showed better results in terms of fid score and newly proposed metric which focus on the face symmetry compared to some of the state-of-the-art methods including lama.
Our paper is availabe at:
https://www.researchgate.net/publication/366984165_SFI-Swin_Symmetric_Face_Inpainting_with_Swin_Transformer_by_Distinctly_Learning_Face_Components_Distributions

The code also will be published in:
https://github.com/mohammadrezanaderi4/SFI-Swin

I can't run the model, I keep getting the error. NameError: name 'normalize_2nd_moment' is not defined

hello,it's amazing.Thank you for your sharing.But when I download your pretrain model,it opens a document.Could you please share your pretrain model with Google drive.Thank you. Besides that, my computer has only one GPU, how do I train the network? Can I just modify the parameters of the number of GPUs trained?

Can you please create new CoreML model ？

HI @ningyu1991

thanks for your work, I have a question with source code in training/networks.py line 246. mask is img[:, -1].unsqueeze(1)
in training/training_loop.py img is img=torch.cat([0.5 - mask, erased_img], dim=1) in training/training_loop.py line 262

so I'm a bit confused about this.

I am getting 404 error in drive

Thank you for open-sourcing your code and pre-training model. I maintain an inpainting tool Lama Cleaner that allows anyone to easily use the SOTA inpainting model. If anyone is interested, you can check out the project's repo.

	LaMa	FcF

Hi, Author, I trained the 512 models (#17 ) but the model didn't converge until now. Since I only have 4 GPUs, I have to use a batch size of 16 or 8. I tried the lr from 0.0005 to 0.002, and different number of training image (--kimg 25000). But the training is unstable and I cannot get a converged model (i.e., FID score is from high to low and stable). Can you be kind to give some advice to handle this? Thank you very much!

Best,
Mike

Thanks for making your great work available as a demo too, when trying to run it on a custom image I get the following error:
RuntimeError: "unfolded2d_copy" not implemented for 'Half'

This project is truly amazing, and I've been trying to export the model for deployment using torch.jit.trace, but I've run into some issues. Could you please help me refine this part of the code? Thank you very much.

RuntimeError:
Could not export Python function call 'BiasActCuda'. Remove calls to Python functions before export. Did you forget to add @script or @script_method annotation? If this is a nn.ModuleList, add it to __constants__:

Traceback (most recent call last):
File "tools/gen_random_segm_masks.py", line 13, in
from metrics.evaluation.masks.mask import SegmentationMask, propose_random_square_crop
ModuleNotFoundError: No module named 'metrics'

when I use the command "bash tools/prepare_celebahq_evaluation.sh",this error came up. My cuda version is as below:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Nov_30_19:08:53_PST_2020
Cuda compilation tools, release 11.2, V11.2.67
Build cuda_11.2.r11.2/compiler.29373293_0

can i use 1024 resolution when infer? and i want to know if i should train a model with 1024X1024 input?

Hello, I encountered this error when running train.py.

Traceback (most recent call last): File "train.py", line 523, in <module> main() # pylint: disable=no-value-for-parameter File "/root/miniconda3/envs/myconda/lib/python3.8/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/root/miniconda3/envs/myconda/lib/python3.8/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/root/miniconda3/envs/myconda/lib/python3.8/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/root/miniconda3/envs/myconda/lib/python3.8/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/root/miniconda3/envs/myconda/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "train.py", line 516, in main subprocess_fn(rank=0, args=args, temp_dir=temp_dir) File "train.py", line 391, in subprocess_fn training_loop.training_loop(rank=rank, **args) File "/FcF-Inpainting-main/training/training_loop.py", line 327, in training_loop loss.accumulate_gradients(phase=phase.name, erased_img=erased_img, real_img=real_img, mask=mask, real_c=real_c, gen_c=gen_c, sync=sync, gain=gain) File "/FcF-Inpainting-main/training/losses/loss.py", line 65, in accumulate_gradients gen_img, _ = self.run_G(g_inputs, gen_c, sync=sync) # May get synced by Gpl. TypeError: run_G() missing 1 required positional argument: 'c'

The second is that I can't download the place2 dataset through wget.

`(myconda) root@PVGa1O:/FcF-Inpainting-main# wget http://data.csail.mit.edu/places/places365/train_large_places365challenge.tar
--2023-02-28 08:34:03-- http://data.csail.mit.edu/places/places365/train_large_places365challenge.tar
Resolving data.csail.mit.edu (data.csail.mit.edu)... 128.52.129.40
Connecting to data.csail.mit.edu (data.csail.mit.edu)|128.52.129.40|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-02-28 08:34:04 ERROR 404: Not Found.

(myconda) root@PVGa1O:/FcF-Inpainting-main# wget http://data.csail.mit.edu/places/places365/val_large.tar
--2023-02-28 08:34:18-- http://data.csail.mit.edu/places/places365/val_large.tar
Resolving data.csail.mit.edu (data.csail.mit.edu)... 128.52.129.40
Connecting to data.csail.mit.edu (data.csail.mit.edu)|128.52.129.40|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-02-28 08:34:19 ERROR 404: Not Found.

(myconda) root@PVGa1O:/FcF-Inpainting-main#
`

hello,,I found that fcf sometimes may generate unreasonable structure in real-world img, and i found that it seems to copy some structures around mask region, could you please give some advices?

Hello author, when you compared other baseline models, did you use the mask generated in this paper and the corresponding data set to retrain the baseline model? Looking forward to your reply！

When I run
python train.py
--outdir=training-runs-inp
--img_data=/raid/yiptmp/zyfai_docker_project/Palette-Image-to-Image-Diffusion-Models/forInpait_512/train
--gpus 2
--resolution 512
--kimg 25000
--gamma 10
--aug 'noaug'
--metrics True
--eval_img_data=/raid/yiptmp/zyfai_docker_project/Palette-Image-to-Image-Diffusion-Models/forInpait_512/random_val
--batch 64

I got this issue,how can I solve it?Thanks!

Hello
How are you?
Thanks for contributing to this project.
The Lama model was trained with 256x256 size of image but it is robust for higher resolution images (ex: 2900x1800) by the with refinement process (https://arxiv.org/pdf/2206.13644.pdf).
I'm curious if this FcF method can be applied for high-resolution input as 2900x1800.
If the image resolution is very high as 2900x1800 and the hole mask size is also bigger than 256x256 or 512x512, how can this method be applied to such images?
Is it possible to integrate such a refinement process like in Lama for high resolution image?

Hello.Thank you for sharing the codes! When I try to download CelebA-HQ Dataset, the link fails. Could you please update the link with Google drive or Baidu drive. Thanks again!

I use --resume places.pkl to finetune on my own datasets, but the result is not good, even worse than places.pkl,,,,i want to know if should i change the lr or other params?

When I run train.py, this error occurs. StyleGAN2Loss in loss.py, line 45.

 with misc.ddp_sync(self.G_synthesis, sync):
            img = self.G_synthesis(x_global, feats, ws)

But G_synthesis need four parameters to run forward. There's no parameter mask here.
Did I get something wrong?

您好，如果使用自制数据集的话也需要分割掩码吗？这个掩码分割的作用是什么呢？

Hello, I encountered this error when running bash tools/prepare_places_segm_evaluation.sh

Detectron v2 is not installed
Traceback (most recent call last):
File "tools/gen_random_segm_masks.py", line 123, in
main(aparser.parse_args())
File "tools/gen_random_segm_masks.py", line 103, in main
process_images(in_files, args.indir, args.outdir, config)
File "tools/gen_random_segm_masks.py", line 28, in process_images
mask_generator = SegmentationMask(**config.mask_generator_kwargs)
AttributeError: 'EasyDict' object has no attribute 'mask_generator_kwargs'

Hello author! Thanks for your wonderful work! I would like to ask how many epochs of training the model on celebahq can converge when the equipment is limited? I only have one gpu, so the training time is very important for me.
Looking forward to your reply！

Thanks for your great work,I encountered this problem while training using the '--augpipe=geom' parameter,i need some help about scaling the training images,thanks for your reply.

python train.py --outdir=training-runs-inp
--img_data=datasets/demo/train
--gpus 4
--kimg 2000
--gamma 10
--metrics True
--eval_img_data datasets/demo/evaluation/random_val
--batch 8
--resolution 512
--augpipe geom

Hello, thank you very much for your contribution. I encountered the following error when I was debugging. When I run the bash tools/prepare_places_val.sh command I get No module named 'training'

Thank you very much for your reply！！！

When i try to train fcf with my custom datasets, I'm facing the problem. Any ideas?

Dear authors,

I tried your Colab demo and ran into a problem. When I tried to inpaint the following image:

with this mask:

I got quite a good result:

However, if I just made the input image brighter (multiplied pixel values by 2):

the inpainting (with the same mask) failed:

I thought it could be some overflow in the image domain, but when I tried to crop the input image a bit (about 10 % on the top and right side):

the inpainting worked again:

Could be some overflow in the frequency domain or am I just doing something wrong? Thank you for your response.

Hi, this is a great job for inpainting. I am training my own dataset, but I didn't find the codes for the loss function described in the paper. Can you please tell where I can find it? Thank you very much！

when I runing bash tools/prepare_places_segm_evaluation.sh

AttributeError: 'EasyDict' object has no attribute 'mask_generator_kwargs'

Hello
How are you?
Thanks for contributing to this project.
I am going to train a 512x512 model on my own data.
But I met the following issue.

I set the resolution parameter to 512 in the script "train.py"

Thanks for your excellent work, but I have encountered some problems.
I got an error when using multiple GPUs:
Traceback (most recent call last):
File "/anaconda3/envs/pytorch_1.10/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap fn(i, *args)
File "/nfs/PycharmProjects/FcF-Inpainting/train.py", line 393, in subprocess_fn training_loop.training_loop(rank=rank, **args)
File "/nfs/PycharmProjects/FcF-Inpainting/training/training_loop.py", line 327, in training_loop
loss.accumulate_gradients(phase=phase.name, erased_img=erased_img, real_img=real_img, mask=mask, real_c=real_c, gen_c=gen_c, sync=sync, gain=gain)
File "/nfs/PycharmProjects/FcF-Inpainting/training/losses/loss.py", line 65, in accumulate_gradients gen_img, _ =self.run_G(g_inputs, mask, gen_c, sync=sync) # May get synced by Gpl.
File "/nfs/PycharmProjects/FcF-Inpainting/training/losses/loss.py", line 44, in run_G
ws[:, cutoff:] = self.G_mapping(torch.randn_like(z), c, skip_w_avg_update=True)[:, cutoff:]
RuntimeError: Output 0 of SliceBackward0 is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.
But when I only run on one GPU, everything is fine.
How should I solve it?

Hi, authors. I used this repository to train with an input image size of 256x256, it worked very well and can converge when I looked at the FID score. However, after I changed the resolution from 256 to 512 in the corresponding files and keep other hyperparameters remained, I have to reduce the batch size for training (from 64 to 8), but the FID score is up during the training. Can you tell me how to handle this? Thank you very much!

Best,
Mike

Hello author, I would like to ask if the Fourier Coarse-to-Fine (FcF) Generator generation network is the class Generator in network.py in the traning folder. I want to try to put this generation network into my model. thank you.

Thanks for your amazing work. And I have some questions for your help:

1. how long time have you trained with celeba-hq?
2. In paper I see you trained 25M images, and the number of Celebahq is 30000, did this mean that you train the model with 800+ epochs?
3. Can I change my inference image's size with model trained with 256*256? Do you have any suggestions if my test images have different size?
   Look forward to your favourable reply~