dome272 / vqgan-pytorch

Pytorch implementation of VQGAN (Taming Transformers for High-Resolution Image Synthesis) (https://arxiv.org/pdf/2012.09841.pdf)

Note:

Code Tutorial + Implementation Tutorial

VQGAN

Vector Quantized Generative Adversarial Networks (VQGAN) is a generative model for image modeling. It was introduced in Taming Transformers for High-Resolution Image Synthesis. The concept is build upon two stages. The first stage learns in an autoencoder-like fashion by encoding images into a low-dimensional latent space, then applying vector quantization by making use of a codebook. Afterwards, the quantized latent vectors are projected back to the original image space by using a decoder. Encoder and Decoder are fully convolutional. The second stage is learning a transformer for the latent space. Over the course of training it learns which codebook vectors go along together and which not. This can then be used in an autoregressive fashion to generate before unseen images from the data distribution.

Results for First Stage (Reconstruction):

1. Epoch:

50. Epoch:

Results for Second Stage (Generating new Images):

Original Left | Reconstruction Middle Left | Completion Middle Right | New Image Right

1. Epoch:

100. Epoch:

Note: Let the model train for even longer to get better results.

---

Train VQGAN on your own data:

Training First Stage

1. (optional) Configure Hyperparameters in training_vqgan.py
2. Set path to dataset in training_vqgan.py
3. python training_vqgan.py

Training Second Stage

1. (optional) Configure Hyperparameters in training_transformer.py
2. Set path to dataset in training_transformer.py
3. python training_transformer.py

Citation

@misc{esser2021taming,
      title={Taming Transformers for High-Resolution Image Synthesis}, 
      author={Patrick Esser and Robin Rombach and Björn Ommer},
      year={2021},
      eprint={2012.09841},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

vqgan-pytorch's Issues

real image in save results looks strange

in the train_vqgan.py, the code real_fake_images = torch.cat((imgs[:4], decoded_images.add(1).mul(0.5)[:4])) should be revised real_fake_images = torch.cat((imgs.add(1).mul(0.5)[:4], decoded_images.add(1).mul(0.5)[:4]))

Discriminator loss mostly zero - it shouldn't be right?

Hi, my discriminator loss curve looks like this:

I don't think that's right because it means most of the times the discriminator knows the generated image is fake but the generator failed to generate better images...
Does anyone have similar issue?

Some questions to produce good generation results.

Hi, thank you very much for sharing your clean implementation. I would like to ask you some questions:

1. How much data we need to produce a good generation result.
2. Does the data need to be of the same category (e.g. flowers).
3. Does transformer require pre-training.

Looking forward for your reply, thanks in advance.

Can you help me with the link of pretrained model

This is a great Repo

I just want to say this is a great repo! I spent two days tuning the environment of the original taming-transformer repo but fails with incompatibility of different packages. But this one is so neat and clean with very basic dependencies. I successfully run it the first time I try. Thank you the repo owner!!!

Best,
Dayang

Conditional image generation problem

Hello, your tutorial is great! I have a question I would like to ask, how do I add conditions in the form of pictures when training the second stage transformer

Grid artifacts in vqgan

Hi, when I run this code I got vertical and horizontal lines on the image:

I came to this repo from the youtube video, and it seems the OP has diagonal lines instead...
https://youtu.be/_Br5WRwUz_U?t=1637
Can anyone tell me why, and how to prevent it?

Black border

Hi! Thank you for sharing this project and for the video tutorial on writing it! I have a question: why do the generated images all have a black border? Can this be fixed? I tried to add "reflect" to all Conv2d layers but that didn't fix it.

Environment Setup

Thanks for the clean and nice implementation of VQGAN.
Could you please provide a requirements.txt or environment.yaml file to set up the environment?
Thanks.

add label conditions problem

Hello, your tutorial is great! I have a question I would like to ask: when I want to add label conditions, such as gender or age, or both, how should the transformer be adjusted, and could you provide an example?

conditional image generation

Hello, your tutorial is great! I have a question I would like to ask, how do I add conditions in the form of pictures when training the second stage transformer

download vqgan_last_ckpt.pt

I have d question that is there anywhere i can donwload the vqgan_last_ckpt.pt?

The issue with training results.

Hello, thank you very much for your code and videos!
I'm using this code repository to train on the flowers dataset with a batch size of 32 for 200 epochs, but the reconstructed images still only have rough outlines without specific details.
Is there something wrong somewhere?

Why there is no valid step when training a VQGAN

In training_vqgan.py, there is no valid step. Is the model trained with this setup usable?

How to distinguish sos token(default = 0) and quantified image token zero ?

Thanks for your video.
Since the transformer take in the quantified image token generated by VQGAN, which codebook has indices (0~n_embed-1), and transformer’s sos token is also set to zero defaultly. Could you tell me why we don't distinguish codebook vector0 and the sos token when training transformer?

How to generate the new image after training the transformer?

I try to train the Transformers, and I get this errors: RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

I try to train the Transformers, and I get this errors: RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

Does transformer forward only predict the last index, z_i? If so, how to obtain the sequence of z_q?

I didn't find how to predict the whole sequence of z_q in the training step. It's not matched with the $p(s) = \prod_ip(s_i|s_{< i})$ in the paper