shi-labs / fcf-inpainting Goto Github PK
View Code? Open in Web Editor NEW[WACV 2023] Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand
Home Page: https://praeclarumjj3.github.io/fcf-inpainting/
License: Other
[WACV 2023] Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand
Home Page: https://praeclarumjj3.github.io/fcf-inpainting/
License: Other
Dear reasercher, please also consider checking our newly introduced face inpainting method to address the symmetry problems of general inpainting mehthods by using swin transformer and semantic aware discriminators.
Our proposed method showed better results in terms of fid score and newly proposed metric which focus on the face symmetry compared to some of the state-of-the-art methods including lama.
Our paper is availabe at:
https://www.researchgate.net/publication/366984165_SFI-Swin_Symmetric_Face_Inpainting_with_Swin_Transformer_by_Distinctly_Learning_Face_Components_Distributions
The code also will be published in:
https://github.com/mohammadrezanaderi4/SFI-Swin
I can't run the model, I keep getting the error. NameError: name 'normalize_2nd_moment' is not defined
hello,it's amazing.Thank you for your sharing.But when I download your pretrain model,it opens a document.Could you please share your pretrain model with Google drive.Thank you. Besides that, my computer has only one GPU, how do I train the network? Can I just modify the parameters of the number of GPUs trained?
Can you please create new CoreML model ?
HI @ningyu1991
thanks for your work, I have a question with source code in training/networks.py line 246. mask is img[:, -1].unsqueeze(1)
in training/training_loop.py img is img=torch.cat([0.5 - mask, erased_img], dim=1) in training/training_loop.py line 262
so I'm a bit confused about this.
I am getting 404 error in drive
Thank you for open-sourcing your code and pre-training model. I maintain an inpainting tool Lama Cleaner that allows anyone to easily use the SOTA inpainting model. If anyone is interested, you can check out the project's repo.
LaMa | FcF | |
---|---|---|
![]() |
![]() |
![]() |
Hi, Author, I trained the 512 models (#17 ) but the model didn't converge until now. Since I only have 4 GPUs, I have to use a batch size of 16 or 8. I tried the lr from 0.0005 to 0.002, and different number of training image (--kimg 25000). But the training is unstable and I cannot get a converged model (i.e., FID score is from high to low and stable). Can you be kind to give some advice to handle this? Thank you very much!
Best,
Mike
Thanks for making your great work available as a demo too, when trying to run it on a custom image I get the following error:
RuntimeError: "unfolded2d_copy" not implemented for 'Half'
This project is truly amazing, and I've been trying to export the model for deployment using torch.jit.trace, but I've run into some issues. Could you please help me refine this part of the code? Thank you very much.
RuntimeError:
Could not export Python function call 'BiasActCuda'. Remove calls to Python functions before export. Did you forget to add @script or @script_method annotation? If this is a nn.ModuleList, add it to __constants__:
Traceback (most recent call last):
File "tools/gen_random_segm_masks.py", line 13, in
from metrics.evaluation.masks.mask import SegmentationMask, propose_random_square_crop
ModuleNotFoundError: No module named 'metrics'
when I use the command "bash tools/prepare_celebahq_evaluation.sh",this error came up. My cuda version is as below:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Nov_30_19:08:53_PST_2020
Cuda compilation tools, release 11.2, V11.2.67
Build cuda_11.2.r11.2/compiler.29373293_0
can i use 1024 resolution when infer? and i want to know if i should train a model with 1024X1024 input?
Hello, I encountered this error when running train.py.
Traceback (most recent call last): File "train.py", line 523, in <module> main() # pylint: disable=no-value-for-parameter File "/root/miniconda3/envs/myconda/lib/python3.8/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/root/miniconda3/envs/myconda/lib/python3.8/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/root/miniconda3/envs/myconda/lib/python3.8/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/root/miniconda3/envs/myconda/lib/python3.8/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/root/miniconda3/envs/myconda/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "train.py", line 516, in main subprocess_fn(rank=0, args=args, temp_dir=temp_dir) File "train.py", line 391, in subprocess_fn training_loop.training_loop(rank=rank, **args) File "/FcF-Inpainting-main/training/training_loop.py", line 327, in training_loop loss.accumulate_gradients(phase=phase.name, erased_img=erased_img, real_img=real_img, mask=mask, real_c=real_c, gen_c=gen_c, sync=sync, gain=gain) File "/FcF-Inpainting-main/training/losses/loss.py", line 65, in accumulate_gradients gen_img, _ = self.run_G(g_inputs, gen_c, sync=sync) # May get synced by Gpl. TypeError: run_G() missing 1 required positional argument: 'c'
The second is that I can't download the place2 dataset through wget.
`(myconda) root@PVGa1O:/FcF-Inpainting-main# wget http://data.csail.mit.edu/places/places365/train_large_places365challenge.tar
--2023-02-28 08:34:03-- http://data.csail.mit.edu/places/places365/train_large_places365challenge.tar
Resolving data.csail.mit.edu (data.csail.mit.edu)... 128.52.129.40
Connecting to data.csail.mit.edu (data.csail.mit.edu)|128.52.129.40|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-02-28 08:34:04 ERROR 404: Not Found.
(myconda) root@PVGa1O:/FcF-Inpainting-main# wget http://data.csail.mit.edu/places/places365/val_large.tar
--2023-02-28 08:34:18-- http://data.csail.mit.edu/places/places365/val_large.tar
Resolving data.csail.mit.edu (data.csail.mit.edu)... 128.52.129.40
Connecting to data.csail.mit.edu (data.csail.mit.edu)|128.52.129.40|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-02-28 08:34:19 ERROR 404: Not Found.
(myconda) root@PVGa1O:/FcF-Inpainting-main#
`
hello,,I found that fcf sometimes may generate unreasonable structure in real-world img, and i found that it seems to copy some structures around mask region, could you please give some advices?
Hello author, when you compared other baseline models, did you use the mask generated in this paper and the corresponding data set to retrain the baseline model? Looking forward to your reply!
When I run
python train.py
--outdir=training-runs-inp
--img_data=/raid/yiptmp/zyfai_docker_project/Palette-Image-to-Image-Diffusion-Models/forInpait_512/train
--gpus 2
--resolution 512
--kimg 25000
--gamma 10
--aug 'noaug'
--metrics True
--eval_img_data=/raid/yiptmp/zyfai_docker_project/Palette-Image-to-Image-Diffusion-Models/forInpait_512/random_val
--batch 64
I got this issue,how can I solve it?Thanks!
Hello
How are you?
Thanks for contributing to this project.
The Lama model was trained with 256x256 size of image but it is robust for higher resolution images (ex: 2900x1800) by the with refinement process (https://arxiv.org/pdf/2206.13644.pdf).
I'm curious if this FcF method can be applied for high-resolution input as 2900x1800.
If the image resolution is very high as 2900x1800 and the hole mask size is also bigger than 256x256 or 512x512, how can this method be applied to such images?
Is it possible to integrate such a refinement process like in Lama for high resolution image?
I use --resume places.pkl to finetune on my own datasets, but the result is not good, even worse than places.pkl,,,,i want to know if should i change the lr or other params?
When I run train.py, this error occurs. StyleGAN2Loss in loss.py, line 45.
with misc.ddp_sync(self.G_synthesis, sync):
img = self.G_synthesis(x_global, feats, ws)
But G_synthesis need four parameters to run forward. There's no parameter mask here.
Did I get something wrong?
您好,如果使用自制数据集的话也需要分割掩码吗?这个掩码分割的作用是什么呢?
Hello, I encountered this error when running bash tools/prepare_places_segm_evaluation.sh
Detectron v2 is not installed
Traceback (most recent call last):
File "tools/gen_random_segm_masks.py", line 123, in
main(aparser.parse_args())
File "tools/gen_random_segm_masks.py", line 103, in main
process_images(in_files, args.indir, args.outdir, config)
File "tools/gen_random_segm_masks.py", line 28, in process_images
mask_generator = SegmentationMask(**config.mask_generator_kwargs)
AttributeError: 'EasyDict' object has no attribute 'mask_generator_kwargs'
Hello author! Thanks for your wonderful work! I would like to ask how many epochs of training the model on celebahq can converge when the equipment is limited? I only have one gpu, so the training time is very important for me.
Looking forward to your reply!
Thanks for your great work,I encountered this problem while training using the '--augpipe=geom' parameter,i need some help about scaling the training images,thanks for your reply.
python train.py --outdir=training-runs-inp
--img_data=datasets/demo/train
--gpus 4
--kimg 2000
--gamma 10
--metrics True
--eval_img_data datasets/demo/evaluation/random_val
--batch 8
--resolution 512
--augpipe geom
Dear authors,
I tried your Colab demo and ran into a problem. When I tried to inpaint the following image:
with this mask:
I got quite a good result:
However, if I just made the input image brighter (multiplied pixel values by 2):
the inpainting (with the same mask) failed:
I thought it could be some overflow in the image domain, but when I tried to crop the input image a bit (about 10 % on the top and right side):
the inpainting worked again:
Could be some overflow in the frequency domain or am I just doing something wrong? Thank you for your response.
Hi, this is a great job for inpainting. I am training my own dataset, but I didn't find the codes for the loss function described in the paper. Can you please tell where I can find it? Thank you very much!
when I runing bash tools/prepare_places_segm_evaluation.sh
AttributeError: 'EasyDict' object has no attribute 'mask_generator_kwargs'
raceback (most recent call last):
File "train.py", line 523, in <module>
main() # pylint: disable=no-value-for-parameter
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "train.py", line 516, in main
subprocess_fn(rank=0, args=args, temp_dir=temp_dir)
File "train.py", line 391, in subprocess_fn
training_loop.training_loop(rank=rank, **args)
File "/content/FcF-Inpainting/training/training_loop.py", line 181, in training_loop
G = dnnlib.util.construct_class_by_name(**G_kwargs, **common_kwargs).train().requires_grad_(False).to(device) # subclass of torch.nn.Module
File "/content/FcF-Inpainting/dnnlib/util.py", line 289, in construct_class_by_name
return call_func_by_name(*args, func_name=class_name, **kwargs)
File "/content/FcF-Inpainting/dnnlib/util.py", line 282, in call_func_by_name
func_obj = get_obj_by_name(func_name)
File "/content/FcF-Inpainting/dnnlib/util.py", line 275, in get_obj_by_name
module, obj_name = get_module_from_obj_name(name)
File "/content/FcF-Inpainting/dnnlib/util.py", line 246, in get_module_from_obj_name
importlib.import_module(module_name) # may raise ImportError
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 724, in exec_module
File "<frozen importlib._bootstrap_external>", line 860, in get_code
File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/content/FcF-Inpainting/training/networks.py", line 1
# Modified from https://github.com/NVlabs/stylegan2-ada-pytorch
^
SyntaxError: invalid character in identifier
Thanks for your excellent work, but I have encountered some problems.
I got an error when using multiple GPUs:
Traceback (most recent call last):
File "/anaconda3/envs/pytorch_1.10/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap fn(i, *args)
File "/nfs/PycharmProjects/FcF-Inpainting/train.py", line 393, in subprocess_fn training_loop.training_loop(rank=rank, **args)
File "/nfs/PycharmProjects/FcF-Inpainting/training/training_loop.py", line 327, in training_loop
loss.accumulate_gradients(phase=phase.name, erased_img=erased_img, real_img=real_img, mask=mask, real_c=real_c, gen_c=gen_c, sync=sync, gain=gain)
File "/nfs/PycharmProjects/FcF-Inpainting/training/losses/loss.py", line 65, in accumulate_gradients gen_img, _ =self.run_G(g_inputs, mask, gen_c, sync=sync) # May get synced by Gpl.
File "/nfs/PycharmProjects/FcF-Inpainting/training/losses/loss.py", line 44, in run_G
ws[:, cutoff:] = self.G_mapping(torch.randn_like(z), c, skip_w_avg_update=True)[:, cutoff:]
RuntimeError: Output 0 of SliceBackward0 is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.
But when I only run on one GPU, everything is fine.
How should I solve it?
Hi, authors. I used this repository to train with an input image size of 256x256, it worked very well and can converge when I looked at the FID score. However, after I changed the resolution from 256 to 512 in the corresponding files and keep other hyperparameters remained, I have to reduce the batch size for training (from 64 to 8), but the FID score is up during the training. Can you tell me how to handle this? Thank you very much!
Best,
Mike
Hello author, I would like to ask if the Fourier Coarse-to-Fine (FcF) Generator generation network is the class Generator in network.py in the traning folder. I want to try to put this generation network into my model. thank you.
Thanks for your amazing work. And I have some questions for your help:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.