advimman / cips

Official repository for the paper "Image Generators with Conditionally-Independent Pixel Synthesis" (CVPR2021, Oral)

CIPS -- Official Pytorch Implementation

of the paper Image Generators with Conditionally-Independent Pixel Synthesis

Requirements

pip install -r requirements.txt

Usage

First create lmdb datasets:

python prepare_data.py images --out LMDB_PATH --n_worker N_WORKER --size SIZE1,SIZE2,SIZE3,... DATASET_PATH

This will convert images to jpeg and pre-resizes it.

To train on FFHQ-256 or churches please run:

python3 -m torch.distributed.launch --nproc_per_node=8 --master_port=1234 train.py --n_sample=8 --batch=4 --fid_batch=8 --Generator=CIPSskip --output_dir=skip-[ffhq/churches] --img2dis --num_workers=16 DATASET_PATH

To train on patches add --crop=PATCH_SIZE. PATCH_SIZE has to be a power of 2.

Pretrained Checkpoints

churches

ffhq256

ffhq1024

landscapes

Generate samples

To play with the models please download checkpoints and check out a notebook.ipynb

Progressive training

We also tried to train progressively on FFHQ starting from 256×256 initialization and got FID 10.07. We will update the paper with the training details soon. Checkpoint name is ffhq1024.pt. Samples are below.

Citation

If you found our work useful, please don't forget to cite

@article{anokhin2020image,
  title={Image Generators with Conditionally-Independent Pixel Synthesis},
  author={Anokhin, Ivan and Demochkin, Kirill and Khakhulin, Taras and Sterkin, Gleb and Lempitsky, Victor and Korzhenkov, Denis},
  journal={arXiv preprint arXiv:2011.13775},
  year={2020}
}

The code is heavely based on the styleganv2 pytorch implementation

Nvidia-licensed CUDA kernels (fused_bias_act_kernel.cu, upfirdn2d_kernel.cu) is for non-commercial use only.

cips's Issues

Checkpoint for Ablation Study?

Hi,

Really nice work! I wonder if you have checkpoints for models in the ablation study? For example, models trained without the coordinate embedding? THanks!

Would you please share discriminator's weight also?

Dear authors:

Hello, thank you for sharing the great project.

I would like to use your project for the baseline of my future research.

Would you please share your discriminator's weight also?

Thank you.

Best,
chkimmmmm

The trained model you provide is not the same as your code

ModSiren Generator

Hello,

Thanks for the code! Trying to run my first experiment but it seems that the default value for the generator is not available. I believe right now the choice is between CIPSskip and CIPSres.

The train.py command works with the above options.

About the blur kernel that affect pixel indenpence

I see that in the code for CIPS block, you use a Blur layer, which use 3x3 blur kernel so I think one pixel is affect by 8 surrouding pixel, is it correct ?

arg.path misdefined?

It seems you are passing the path argument to args.out_path (line 332 in train.py). Is that the normal procedure?

Should that be commented out, in order to use the path argument in Multiscale dataset? Or is it expected that the dataset folders are inside the output_path?

Thanks!

Learning & feeding in coordinate embeddings for Landscape panoramas

Hi, excellent work!

In your paper, you discuss a Landscapes dataset of FLICKR images that are randomly cropped to 256x256 for training. Figure 15 also shows results for patch-based training on Churches and FFHQ. How many coordinate embeddings are learned in the patch-based training settings? Are there patch_height x patch_width learned embeddings, or full_res_image_height x full_res_image_width embeddings? If the former, do you tile the patch embedding grids when synthesizing panoramas?

I tried to train with your code,but it didn't work.The main progress seems to stay waiting for something to load.

/home/hello/miniconda3/envs/deepsort/bin/python3 /home/hello/桌面/CIPS-main/train.py LMDB_data1024*1024_anime --nproc_per_node=8 --master_port=1234 --n_sample=8 --batch=4 --fid_batch=8 --Generator=CIPSskip --output_dir=skip-[ffhq/churches] --img2dis --num_workers=16
Traceback (most recent call last):
File "/home/hello/桌面/CIPS-main/train.py", line 16, in
import model
File "/home/hello/桌面/CIPS-main/model/init.py", line 1, in
from .Discriminators import *
File "/home/hello/桌面/CIPS-main/model/Discriminators.py", line 8, in
from .blocks import ConvLayer, ResBlock, EqualLinear
File "/home/hello/桌面/CIPS-main/model/blocks.py", line 8, in
from op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
File "/home/hello/桌面/CIPS-main/op/init.py", line 2, in
from .upfirdn2d import upfirdn2d
File "/home/hello/桌面/CIPS-main/op/upfirdn2d.py", line 13, in
os.path.join(module_path, 'upfirdn2d_kernel.cu'),
File "/home/hello/miniconda3/envs/deepsort/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1092, in load
keep_intermediates=keep_intermediates)
File "/home/hello/miniconda3/envs/deepsort/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1307, in _jit_compile
baton.wait()
File "/home/hello/miniconda3/envs/deepsort/lib/python3.7/site-packages/torch/utils/file_baton.py", line 42, in wait
time.sleep(self.wait_seconds)
KeyboardInterrupt

Process finished with exit code 1

parameters

Thanks for sharing nice implementation!
Could you explain how is the initialization constant being chosen in LFF block?

https://github.com/saic-mdal/CIPS/blob/eadae6e45d8c1f3657faa88a065b59990747cd16/model/blocks.py#L517

What does lr_mlp in CIPSskip stands for?

def __init__(self, size=256, hidden_size=512, n_mlp=8, style_dim=512, lr_mlp=0.01,

Quick question about training time

Hello authors,

Thank you for your paper -- it looks very interesting! I had a quick question about training time: how long (and with how many GPUs) does it take you to train the pretrained models provided in the repo? Do you find that the method is slow or fast relative to StyleGAN2?

I look forward to playing around with the repo over the next week!

Best,
greeneggsandyaml

Is this code compatible with RTX 3090, CUDA Version 11 ?

train model

Why do you have an generator and g_ema in your training code？And Your optimizer optimizes generator directly，why not optimizes g_ema? And why accumulate(g_ema, g_module, accum) ，not And why accumulate(g_ema, generator, accum) ？

Timed out when calculating FID

Hi. First of all, thanks for your great work.

I have a problem when calculating FID during training. Every args.save_checkpoint_frequency iterations, the model is evaluated by calculating FID score. However, at this phase, I have a timed out problem. Here is the error log.

[E ProcessGroupNCCL.cpp:566] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802801 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:325] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802801 milliseconds before timing out.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 1 (pid: 4414) of binary: /opt/conda/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed

Before this log came out, there was no change in the prompt but GPUs were still working. As I'm not that familiar with distributed training, I want to ask how to fix this problem. I've tried to lower the number of args.fid_samples, but this doesn't help. Thank you.

Best Wishes,
Lee

Would you share progressive training code?

Dear authors:

Hello.

First of all, thank you for the sharing this great work.

In the Read.me, you mentioned that you are going to share the progressive training details.

When will you share it?

Thank you.

Best.
chkimmmmm

Finding mean for truncation trick

Can you please explain what this does in the notebook? Should this truncation be recomputed if one is to create more diversity of generation?

Satellite-Landscapes dataset

Hi!
Could you please specify, how you mine the Satellite-Landscapes dataset?
Best regards, Andrey.