Woncheol Shin1, Gyubok Lee1, Jiyoung Lee1, Joonseok Lee2,3, Edward Choi1 | Paper
1KAIST, 2Google Research, 3Seoul National University
Recently, vector-quantized image modeling has demonstrated impressive performance on generation tasks such as text-to-image generation. However, we discover that the current image quantizers do not satisfy translation equivariance in the quantized space due to aliasing, degrading performance in the downstream text-to-image generation and image-to-text generation, even in simple experimental setups. Instead of focusing on anti-aliasing, we take a direct approach to encourage translation equivariance in the quantized space. In particular, we explore a desirable property of image quantizers, called 'Translation Equivariance in the Quantized Space' and propose a simple but effective way to achieve translation equivariance by regularizing orthogonality in the codebook embedding vectors. Using this method, we improve accuracy by +22% in text-to-image generation and +26% in image-to-text generation, outperforming the VQGAN.
conda env create -f environment.yaml
conda activate bidalle
pip install horovod==0.22.1
If you fail to install horovod, please refer to here.
bash download_mnist64x64_stage2.sh
bash download_classifier_ckpt.sh
In run_train_dalle.sh
, you should specify --vqgan_model_path
and --vqgan_config_path
.
Provide your model path pretrained from TE-VQGAN.
For example,
--vqgan_model_path /home/TE-VQGAN/logs/2022-04-01T07-37-39_mnist64x64_vqgan/checkpoints/last.ckpt \
--vqgan_config_path /home/TE-VQGAN/logs/2022-04-01T07-37-39_mnist64x64_vqgan/configs/2022-04-01T07-37-39-project.yaml
And then run the script:
bash run_train_dalle.sh
@misc{shin2021translationequivariant,
title={Translation-equivariant Image Quantizer for Bi-directional Image-Text Generation},
author={Woncheol Shin and Gyubok Lee and Jiyoung Lee and Joonseok Lee and Edward Choi},
year={2021},
eprint={2112.00384},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
The implementation of 'TE-VQGAN' and 'Bi-directional Image-Text Generator' is based on VQGAN and DALLE-pytorch.