jerryli27 / twingan

Twin-GAN -- Unpaired Cross-Domain Image Translation with Weight-Sharing GANs

TwinGAN -- Unsupervised Image Translation for Human Portraits

Use Pretrained Model.

We provide two pre-trained models: human to anime and human to cats.

Run the following command to translate the demo inputs.

python inference/image_translation_infer.py \
--model_path="/PATH/TO/MODEL/256/"
--image_hw=256
--input_tensor_name="sources_ph"
--output_tensor_name="custom_generated_t_style_source:0"
--input_image_path="./demo/inference_input/"
--output_image_path="./demo/inference_output/"

The input_image_path can be either one single image or a path containing images.

For more information, see the documentation on inference and eval and on the web interface.

Training

Download CelebA and the Getchu dataset by following the datasets guide. Then train your model using script from the training guide.

Blog and Technical report.

An English blog and a Chinese 中文 blog are published in early April 2018 and are available for readers with less technical background.

Network setup:

Conv layer structure:

Please refer to the technical report for details on the network structure and losses.

Extra materials:

Presentation Slides at Anime Expo 2018

Related works

Our idea of using adaptive normalization parameters for image translation is not unique. To the best of our knowledge, at least two more work have similar ideas: MUNIT and EG-UNIT. Our model is developed around the same time period as these models.

Some key differences between our model and the two mentioned are -- we find UNet to be extremely helpful in maintaining semantic correspondence across domain, and we found that sharing all convolution filter weights speeds up training while maintaining the same output quality.

Documentations

More documentations can be found under docs/

Reference

A lot of the code are adapted from online. Here is a non-exhaustive list of the repos where I borrowed code from extensively.

TF-Slim image models library

PGGAN

Anime related repos and datasets

Shameless self promotion of my AniSeg anime object detection & segmentation model.

Sketch coloring using PaintsTransfer and PaintsChainer.

Create anime portraits at Crypko and MakeGirlsMoe

The all-encompassing anime dataset Danbooru2017 by gwern.

My hand-curated sketch-colored image dataset.

Disclaimer

This personal project is developed and open sourced when I am working for Google, therefore you see Copyright 2018 Google LLC in each file. This is not an officially supported Google product. See License and Contributing for more details.

twingan's Issues

Issue ERROR:tensorflow:Cannot parse hostport

Hello, I have tried to run the web interface sample following the documents on a windows docker environment but have run into this tensorflow issue after launching the website. Am unable to find any relevant ways to resolve in online, could I get a clue on what I may have done wrong?

this only occurred after I set the PYTHONPATH to twinGAN, installed typing and various other modules and then finally launched it.

this are the steps I took to reach this error

//set up docker with tensorflow image
docker run -it -p 8888:8888 -p 6006:6006 -p 9000:9000 -p 8000:8000 -v //c/Users/asceansion/desktop/tgan:/home tensorflow/tensorflow
//set path to allow appropriate access to modules
export PYTHONPATH="$PWD/twingan"
//install correct dependencies
pip install -U -r requirement.txt

//start server to host web demo
python /home/tgan2/twingan/interface/server.py --twingan_server=localhost:9000 --twingan_server=localhost:9000

cd /PATH/TO/TwinGAN/interface
python /home/tgan2/twingan/interface/server.py
--twingan_server=localhost:9000
--image_hw=256
--gpu=1

Test Demo KeyError: 'ClipByValue'

I was trying to run the demo follow the instructions in:
python3.6
tensorflow 1.7.0
MacOS High Sierra version 10.13.6
it gave

python inference/image_translation_infer.py \
--model_path="/Users/kiralpoon/Git/TwinGAN/256/" \
--image_hw=256 \
--input_tensor_name="sources_ph" \
--output_tensor_name="custom_generated_t_style_source:0" \
--input_image_path="./demo/inference_input/" \
--output_image_path="./demo/inference_output/"/Users/kiralpoon/anaconda/envs/tf1.6/lib/python3.6/site-packages/absl/flags/_validators.py:359: UserWarning: Flag --model_path has a non-None default value; therefore, mark_flag_as_required will pass even if flag is not specified in the command line!
  'command line!' % flag_name)
Loading inference model
2018-08-09 13:36:14.032130: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Model directory: /Users/kiralpoon/Git/TwinGAN/256/
Metagraph file: model.ckpt-100000.meta
Checkpoint file: model.ckpt-100000
start assign saver
/Users/kiralpoon/Git/TwinGAN/256/model.ckpt-100000.meta
Traceback (most recent call last):
  File "inference/image_translation_infer.py", line 128, in <module>
    tf.app.run()
  File "/Users/kiralpoon/anaconda/envs/tf1.6/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "inference/image_translation_infer.py", line 103, in main
    inferer = ImageInferer()
  File "inference/image_translation_infer.py", line 61, in __init__
    util_io.load_model(FLAGS.model_path, input_map=input_map)
  File "/Users/kiralpoon/Git/TwinGAN/util_io.py", line 172, in load_model
    saver = tf.train.import_meta_graph(metapath)
  File "/Users/kiralpoon/anaconda/envs/tf1.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1909, in import_meta_graph
    **kwargs)
  File "/Users/kiralpoon/anaconda/envs/tf1.6/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 737, in import_scoped_meta_graph
    producer_op_list=producer_op_list)
  File "/Users/kiralpoon/anaconda/envs/tf1.6/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
    return func(*args, **kwargs)
  File "/Users/kiralpoon/anaconda/envs/tf1.6/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 431, in import_graph_def
    _RemoveDefaultAttrs(op_dict, producer_op_list, graph_def)
  File "/Users/kiralpoon/anaconda/envs/tf1.6/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 211, in _RemoveDefaultAttrs
    op_def = op_dict[node.op]
KeyError: 'ClipByValue'

I also tried to switch to tensorflow 1.6.0 but with no luck.

Here is the ls of the 256 downloaded pre-trained model

ls   
checkpoint                            flags.txt                             graph.pbtxt                           model.ckpt-100000.data-00000-of-00001 model.ckpt-100000.index               model.ckpt-100000.meta

Impressive work! I hope I will be able to run it and contribute.

Some details about the training parameters.

I trained the network from scratch but got poor results.
Below are my training parameters(same as the training guide you provide).

--program_name=twingan
--dataset_name="image_only"
--dataset_dir="dir to celeba tfrecord files"
--unpaired_target_dataset_name="anime_faces"
--unpaired_target_dataset_dir="dir to the anime tfrecord you provided"
--train_dir="dir to save results"
--dataset_split_name=train
--preprocessing_name="danbooru"
--resize_mode=RANDOM_CROP
--do_random_cropping=True
--learning_rate=0.0001
--learning_rate_decay_type=fixed
--is_training=True
--generator_network="pggan"
--loss_architecture=dragan
--pggan_max_num_channels=256
--generator_norm_type=batch_renorm
--use_ttur=True
--num_images_per_resolution=50000

Compared with official PGGAN repo, I found some differences.

1. num_image_per_resolution of TwinGAN is 50000 while PGGAN is 600000.
2. TwinGAN uses RANDOM_CROP while PGGAN uses RESIZE directly.

Could you please help me out here?

Image synthesis through illust2vec.

Thank you in advanced, very helpful article.
In your blog, you mentioned that:

"The model does not support direct modification of such features. I used illust2vec to extract character features such as hair color, eye color, etc. Then I got some ideas from Conditional generative adversarial nets and supplied those features as embeddings to Generator. Now when I generate an image, I add an additional Anime character as input, and the transferred result should look like that character, with the position and facial expression(TODO) of the human portrait kept untouched. The result is like this:"

I wonder is there any pre-trained model of this feature yet?

Regards,
MinakoKojima

Can you share more pretrained model? I want to compare the results.

Can you share more pretrained model? I want to compare the results.

How many pictures is the training dataset required for the model?

If I want to train a picture translation model myself, how to set the size of the data set?
I mean, how many pictures from celeba and how many pictures from the anime data set are required to train the model.

About Getchu Dataset

Hi,
I want to convert anime tf-record files to jpg files for Pytorch use.
I noticed that I can only get the one-hot array with the following code:

def read_and_decode(filename_queue):
    reader = tf.TFRecordReader()
    _, serialized_example = reader.read(filename_queue)
    features = tf.parse_single_example(
        serialized_example,
        # Defaults are not specified since both keys are required.
        features={
            'image/colorspace': tf.FixedLenFeature([], tf.string),
            'image/channels': tf.FixedLenFeature([], tf.int64),
            'image/format': tf.FixedLenFeature([], tf.string),
            'image/filename': tf.FixedLenFeature([], tf.string),
            'image/encoded': tf.FixedLenFeature([], tf.string),
        })
    image = tf.decode_raw(features['image/encoded'], tf.uint8)
    return image

I still need to know the width and height of the array.
Could you please help me with that?

The result of my own training, using the image_translation_infer.py test, looks very bad.

The result of my own training, using the image_translation_infer.py test, looks very bad.

cuDNN version

Hi!

Thank you for your work, it's really cool!
Could you please specify which cuda and cudnn versions you have?

AttributeError: 'NoneType' object has no attribute 'shape'

I got the error when I run pggan_runner.py.
env:
python 2.7
tf-gpu 1.9.0

Traceback (most recent call last):
File "pggan_runner.py", line 164, in
tf.app.run()
File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "pggan_runner.py", line 154, in main
model.main()
File "/root/style_transfer/TwinGAN/image_generation.py", line 1055, in main
super(GanModel, self).main()
File "/root/style_transfer/TwinGAN/model/model_inheritor.py", line 1055, in main
'is_training': FLAGS.is_training})
File "/root/style_transfer/TwinGAN/deployment/model_deploy.py", line 237, in create_clones
outputs = model_fn(*args, **kwargs)
File "/root/style_transfer/TwinGAN/image_generation.py", line 250, in _clone_fn
generator_kwargs = {'is_training': is_training, 'target_shape': targets.shape}
AttributeError: 'NoneType' object has no attribute 'shape'
Thank you~

Name of dataset unknown error on CelebA and Getchu Datasets

Hi!
This is a really cool project, and thank you for sharing it publicly!

I'm trying to retrain your model to get some hands-on experience with GANs, but I'm having a bit of an issue setting up the training task.
I've downloaded the CelebA dataset and ran the celeba_convert.py script on it to pre-process it into tfrecord files, and I also downloaded the getchu dataset you prepared.

I set up a conda env with all the requirements and I created a bash file with the training script you provided and changed the file paths accordingly.
I get the following error, where it says "Name of Dataset unknown". Could it be because of the way I've structured my training script?

Would really appreciate any insight!

python pggan_runner.py
--program_name=twingan
--dataset_name="image_only"
--dataset_dir="data/celeba/"
--unpaired_target_dataset_name="anime_faces"
--unpaired_target_dataset_dir="data/anime_faces/"
--train_dir="checkpoints/twingan_faces/"
--dataset_split_name=train
--preprocessing_name="danbooru"
--resize_mode=RESHAPE
--do_random_cropping=True
--learning_rate=0.0001
--learning_rate_decay_type=fixed
--is_training=True
--generator_network="pggan"
--use_unet=True
--num_images_per_resolution=300000
--loss_architecture=dragan
--gradient_penalty_lambda=0.25
--pggan_max_num_channels=256
--generator_norm_type=batch_renorm
--hw_to_batch_size="{4: 8, 8: 8, 16: 8, 32: 8, 64: 8, 128: 4, 256: 3, 512: 2}"
--do_pixel_norm=True
--l_content_weight=0.1
--l_cycle_weight=1.0

WARNING:tensorflow:Checkpoint for resolution 4 does not exist yet! Falling back to the previous checkpoint.
Traceback (most recent call last):
File "pggan_runner.py", line 164, in
tf.app.run()
File "/home/imachines/anaconda3/envs/twingan/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "pggan_runner.py", line 154, in main
model.main()
File "/home/imachines/Desktop/TwinGAN/TwinGAN/image_generation.py", line 1055, in main super(GanModel, self).main() File "/home/imachines/Desktop/TwinGAN/TwinGAN/model/model_inheritor.py", line 991, in main dataset = self._select_dataset() File "/home/imachines/Desktop/TwinGAN/TwinGAN/image_generation.py", line 203, in _select_dataset dataset = super(GanModel, self)._select_dataset() File "/home/imachines/Desktop/TwinGAN/TwinGAN/model/model_inheritor.py", line 326, in _select_dataset FLAGS.dataset_name, FLAGS.dataset_split_name, FLAGS.dataset_dir) File "/home/imachines/Desktop/TwinGAN/TwinGAN/datasets/dataset_factory.py", line 79, in get_datasetName of dataset unknown %s' % name raise ValueError('Name of dataset unknown %s' % name) ValueError: Name of dataset unknown trainModel.sh: line 4: --program_name=twingan: command not found
trainModel.sh: line 5: --dataset_name=image_only: command not found
trainModel.sh: line 6: --dataset_dir=datasets/celeba/: No such file or directory
trainModel.sh: line 7: --unpaired_target_dataset_name=anime_faces: command not found
trainModel.sh: line 8: --unpaired_target_dataset_dir=datasets/anime_faces/: No such file or directory
trainModel.sh: line 9: --train_dir=checkpoints/twingan_faces/: No such file or directory
trainModel.sh: line 10: --dataset_split_name=train: command not found
trainModel.sh: line 11: --preprocessing_name=danbooru: command not found
trainModel.sh: line 12: --resize_mode=RESHAPE: command not found
trainModel.sh: line 13: --do_random_cropping=True: command not found
trainModel.sh: line 14: --learning_rate=0.0001: command not found
trainModel.sh: line 15: --learning_rate_decay_type=fixed: command not found
trainModel.sh: line 16: --is_training=True: command not found
trainModel.sh: line 17: --generator_network=pggan: command not found
trainModel.sh: line 18: --use_unet=True: command not found
trainModel.sh: line 19: --num_images_per_resolution=300000: command not found
trainModel.sh: line 20: --loss_architecture=dragan: command not found
trainModel.sh: line 21: --gradient_penalty_lambda=0.25: command not found
trainModel.sh: line 22: --pggan_max_num_channels=256: command not found
trainModel.sh: line 23: --generator_norm_type=batch_renorm: command not found
trainModel.sh: line 24: --hw_to_batch_size={4: 8, 8: 8, 16: 8, 32: 8, 64: 8, 128: 4, 256: 3, 512: 2}: command not found
trainModel.sh: line 25: --do_pixel_norm=True: command not found
trainModel.sh: line 26: --l_content_weight=0.1: command not found
trainModel.sh: line 27: --l_cycle_weight=1.0: command not found

Implement in a chat bot

Hello! I found the project to be really interesting, do you mind if I implement it in a chat bot to expose it to more people? Not sure if this is the right place to write it

face crop

Hi,
Thank you for sharing such a fun project！But, I'm sorry that it doesn't work well when I use my images(they're all Asian) and your pretrained model.I guess whether because I didn't put the face right in the center of the image.
Is that right?
Thanks a lot.

Share MeowGAN dataset

Dataset comes from here. Sharing the tfrecords I used will hopefully be useful.

Placeholder issue

When I run as follows:
python inference/image_translation_infer.py --model_path="~/Documents/weights/256" --image_hw=256 --input_tensor_name="sources_ph" --output_tensor_name="custom_generated_t_style_source:0" input_image_path="~/Documents/art/photo.jpg" --output_image_path="~/Documents/art/" --num_output=1
I get that error:
Traceback (most recent call last):
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call
return fn(*args)
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'Placeholder' with dtype uint8 and shape [?,?,?]
[[Node: Placeholder = Placeholderdtype=DT_UINT8, shape=[?,?,?], _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
[[Node: custom_generated_t_style_source/_2051 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1565_custom_generated_t_style_source", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "inference/image_translation_infer.py", line 128, in
tf.app.run()
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "inference/image_translation_infer.py", line 109, in main
outputs, image_paths = inferer.infer(FLAGS.input_image_path, return_image_paths=True, num_output=FLAGS.num_output)
File "inference/image_translation_infer.py", line 91, in infer
output = self.sess.run(self.output, feed_dict=feed_dict)
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 877, in run
run_metadata_ptr)
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
run_metadata)
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'Placeholder' with dtype uint8 and shape [?,?,?]
[[Node: Placeholder = Placeholderdtype=DT_UINT8, shape=[?,?,?], _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
[[Node: custom_generated_t_style_source/_2051 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1565_custom_generated_t_style_source", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'Placeholder', defined at:
File "inference/image_translation_infer.py", line 128, in
tf.app.run()
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "inference/image_translation_infer.py", line 103, in main
inferer = ImageInferer()
File "inference/image_translation_infer.py", line 55, in init
self.images_placeholder = tf.placeholder(dtype=tf.uint8, shape=[None, None, None])
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1735, in placeholder
return gen_array_ops.placeholder(dtype=dtype, shape=shape, name=name)
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 4925, in placeholder
"Placeholder", dtype=dtype, shape=shape, name=name)
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
op_def=op_def)
File "/home/bakirillov/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1717, in init
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'Placeholder' with dtype uint8 and shape [?,?,?]
[[Node: Placeholder = Placeholderdtype=DT_UINT8, shape=[?,?,?], _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
[[Node: custom_generated_t_style_source/_2051 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1565_custom_generated_t_style_source", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
I checked and in inference/image_translation_infer.py everything looks correct.
What am I doing wrong?
Python 3.6.4. Anaconda, tensorflow 1.10, keras 2.2.2

ImportError: No module named 'pggan_utils'

E:\Users\Raytine\Anaconda3\python.exe F:/zhaiyao/TwinGAN-master/pggan_runner.py
Traceback (most recent call last):
File "F:/zhaiyao/TwinGAN-master/pggan_runner.py", line 39, in
import image_generation
File "F:\zhaiyao\TwinGAN-master\image_generation.py", line 40, in
from nets import pggan
File "F:\zhaiyao\TwinGAN-master\nets\pggan.py", line 19, in
from pggan_utils import pggan_utils
ImportError: No module named 'pggan_utils'

Inference model trained on Multiple GPU

For multiple GPU train, the saved model ( in .meta file) is split to two clones.
So all the tensor names are changed by default like
source_ph --> clone_0/ source_ph
custom_generated_t_style_source --> clone_0/custom_generated_t_style_source
So, if anyone want to eval or inference on its own single GPU machine, please be careful when the pre_trained model is base on multiple GPU.
I recommend use this inference code

python inference/image_translation_infer.py \
--model_path="/PATH/TO/CHECHPOINT" \
--image_hw=128 \
--input_tensor_name="clone_0/sources_ph" \
--output_tensor_name="clone_0/custom_generated_t_style_source" \
--input_image_path="/PATH/TO/INPUT" \
--output_image_path="/PATH/TO/OUTPUT" \

And by the way, I am wondering why the inference speed is soooooo slow. Loading the weight take 5-10 second on my 1080Ti

identity preserving?

Thanks for your great work. I'm just wondering how do you measure or evaluate how well the identity is preserved in your results?

At least visually it is hard for me to recognize Emma in the anime results.

Dataset size

Hi,
I would like to try to train the model using CelebA paired with a different dataset I am building.
What should be a good number of images I should aim for obtaining a good result?
If my dataset is way smaller than CelebA, do I need to generate again the CelebA tfrecords using fewer images or it isn't necessary?

How many encoders do you use?

Your paper said that "your TwinGAN architecture uses two encoders, E1 and E2". But in your blog, I saw that "As mentioned in the Facebook paper, letting the human and anime portraits share the same network will help the network realize that......"
So, how many encoders are there in the TwinGAN architecture?

Cosine Distance

Hi, thank you for such a nice piece of work.

In Section 4.4 in your paper, it is mentioned that you have calculated the cosine distance between the flattened embedding. If we have 4 images. i.e. f(x1), f(x1'), and f(x2), f(x2') where f(x1) and f(x1') are the real images and generated images in domain X1 and f(x2) and f(x2') are the real images and generated images in domain X2. Can you please suggest did you calculate the cosine distance between f(x1) and f(x1') or between f(x1) and f(x2') or between f(x1) and f(x2) ?

Thanks in advance.

First Usage Instructions - Training on another dataset

Firstly, I would like to thank you for this amazing work.

I am interested in using your implementation on image-to-image translation (synthetic to real-world data) on my own dataset, for example, GTA to Cityscape as done by the UNIT.

What are the steps of getting your implementation running on my dataset ?

ModuleNotFoundError: No module named 'util_io'

Thank you very much for this project. Unfortunately, I am unable to reproduce the results.
I'm running the script in the following way:
python image_translation_infer.py --model_path="/Documents/weights/256" --image_hw=256 --input_tensor_name="sources_ph" --output_tensor_name="custom_generated_t_style_source:0" --input_image_path="/Documents/art/photo.png" --output_image_path="/Documents/art/"
and get the following error:
Traceback (most recent call last):
File "image_translation_infer.py", line 35, in
import util_io
ModuleNotFoundError: No module named 'util_io'
Could you please look into that?
Python 3.6.4. Anaconda, tensorflow 1.10, keras 2.2.2

Trying to train dataset but getting: [Errno 21] Is a directory: '/'

I'm trying to train my own dataset, but I get stuck on the conversion of images, I'm running the following code:

python3 convert_image_only.py \
--train_directory="/Users/username/Downloads/twina/datasets/faces/" \
--validation_directory="/Users/username/Downloads/twina/datasets/faces/" \
--output_directory="/Users/username/Downloads/twina/datasets/out/" \
--train_shards=8 \
--validation_shards=2 \
--num_threads=2

But I keep getting the following error:

Saving results to /Users/username/Downloads/twina/datasets/out/
Traceback (most recent call last):
  File "convert_image_only.py", line 91, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/lib/python3.7/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "convert_image_only.py", line 87, in main
    converter.main()
  File "/Users/username/Downloads/twina/datasets/datasets/convert_general_image_data.py", line 307, in main
    shared_info = self._get_shared_info()
  File "/Users/username/Downloads/twina/datasets/datasets/convert_general_image_data.py", line 110, in _get_shared_info
    util_io.touch_folder(shared_info_output_dir)
  File "/Users/username/Downloads/twina/datasets/util_io.py", line 41, in touch_folder
    os.makedirs(dn)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
IsADirectoryError: [Errno 21] Is a directory: '/'

i have problem in _get_batch function

could you help me about this problem?
i ran python image_generation.py

this is result message.

Traceback (most recent call last):
  File "image_generation.py", line 1050, in <module>
    tf.app.run()
  File "C:\Users\User\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
    _sys.exit(main(argv))
  File "image_generation.py", line 1046, in main
    model.main()
  File "image_generation.py", line 1041, in main
    super(GanModel, self).main()
  File "C:\Users\User\Desktop\capston\twingan\model\model_inheritor.py", line 1017, in main
    data_batched = self._get_batch(data)
  File "C:\Users\User\Desktop\capston\twingan\model\model_inheritor.py", line 392, in _get_batch
    batch = tf.train.batch(batch_input, batch_size=FLAGS.batch_size, enqueue_many=False, num_threads=FLAGS.num_preprocessing_threads, capacity=5*(FLAGS.batch_size))
  File "C:\Users\User\Anaconda3\lib\site-packages\tensorflow\python\training\input.py", line 988, in batch
    name=name)
  File "C:\Users\User\Anaconda3\lib\site-packages\tensorflow\python\training\input.py", line 746, in _batch
    tensor_list = _validate(tensor_list)
  File "C:\Users\User\Anaconda3\lib\site-packages\tensorflow\python\training\input.py", line 605, in _validate
    tensor_list = ops.convert_n_to_tensor_or_indexed_slices(tensor_list)
  File "C:\Users\User\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1325, in convert_n_to_tensor_or_indexed_slices
    values=values, dtype=dtype, name=name, as_ref=False)
  File "C:\Users\User\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1287, in internal_convert_n_to_tensor_or_indexed_slices
    raise TypeError("values must be a list.")
TypeError: values must be a list.

i printed batch_input, result is below.

dict_values([<tf.Tensor 'Merge:0' shape=(?, ?, 3) dtype=float32>, <tf.Tensor 'Cast:0' shape=(40,) dtype=float32>, <tf.Tensor 'Reshape_4:0' shape=(10,) dtype=float32>, <tf.Tensor 'Reshape_2:0' shape=() dtype=string>])

thanks.

How to use inference/image_translation_infer.py in my trained model?

when i use my model model/128 to replace the model/256 ,

tf.flags.DEFINE_string("model_path", "model/128/", "Path containing a checkpoint.")
tf.flags.DEFINE_integer('image_hw',128, 'height and width of the input image.')
tf.flags.DEFINE_string('input_tensor_name','custom_input_ph', 'Optional input tensor name. e.g. sources_ph.')
tf.flags.DEFINE_string('output_tensor_name','custom_generated_targets', 'e.g. custom_generated_t_style_source:0')
tf.flags.mark_flags_as_required(['model_path', 'output_tensor_name'])
FLAGS = tf.flags.FLAGS

I get an error:
ValueError: Attempted to map inputs that were not found in graph_def: [custom_input_ph:0]

How to use the corresponding encoder to extract the latent embeddings？

For each domain, we use the corresponding encoder to extract the latent embeddings. For each image in domain X1, we find the nearest neighbors in domain X2 by calculating the cosine distances between the flattened embeddings.

How to implement this part with code？
Thank you!

How to use your own dataset: ValueError: Node 'generator_1/latent_vector/Pad' has an _output_shapes attribute inconsistent with the GraphDef for output #0:

I'm making dataset that contains CelebA (provided on the website) and Danbooru Face Dataset, to create Japanese animated characters when a human face inputs.

I visited your github (https://github.com/jerryli27/TwinGAN) and i refer this link to make own dataset (https://github.com/jerryli27/TwinGAN/blob/master/docs/use_your_dataset.md).

python datasets/convert_image_only.py 
 --train_directory="./test/train-A" (Train-A CelebA Dataset(jpg photo))
 --validation_directory="./test/train-B" (Train-B Danbooru Dataset(jpg photo))
 --output_directory="./test/train-B" (Train-B Danbooru Dataset(jpg photo))
 --train_shards=8 --validation_shards=2 --num_threads=2

python image_generation.py 
 --dataset_name="image_only"
 --dataset_dir="./test/train-B" (tfrecord file output directory)
 --dataset_use_target=True --train_dir="./checkpoints/test/" 
 --dataset_split_name=train --learning_rate=0.0001 
 --learning_rate_decay_type=fixed --is_training=True 
 --train_image_size=4 --preprocessing_name="danbooru" 
 --generator_network="pggan" --ignore_missing_vars=True --max_number_of_steps=100000

After Training, I executed python image_translation_infer.py
(I changed image_translation_infer.py like this)

tf.flags.DEFINE_string("model_path", "./checkpoints/test", "Path containing a checkpoint.")
tf.flags.DEFINE_integer('image_hw', 256, 'height and width of the input image.')
tf.flags.DEFINE_string('input_tensor_name', "custom_input_ph", 'Optional input tensor name. e.g. sources_ph.')
tf.flags.DEFINE_string('output_tensor_name', "custom_generated_targets:0", 'e.g. custom_generated_t_style_source:0')

Model directory: ./checkpoints/test
Metagraph file: model.ckpt-50000.meta
Checkpoint file: model.ckpt-50000
Traceback (most recent call last):
  File "image_translation_infer.py", line 95, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "image_translation_infer.py", line 81, in main
    inferer = ImageInferer()
  File "image_translation_infer.py", line 33, in __init__
    util_io.load_model(FLAGS.model_path, input_map=input_map)
  File "/home/itsc/Desktop/TwinGAN/util_io.py", line 168, in load_model
    saver = tf.train.import_meta_graph(os.path.join(model_exp, meta_file), input_map=input_map)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1960, in import_meta_graph
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/meta_graph.py", line 744, in import_scoped_meta_graph
    producer_op_list=producer_op_list)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 422, in import_graph_def
    raise ValueError(str(e))
ValueError: Node 'generator_1/latent_vector/Pad' has an _output_shapes attribute inconsistent with the GraphDef for output #0: Dimension 1 in both shapes must be equal, but are 262 and 7. Shapes are [1,262,262,?] and [?,7,7,256].

ValueError: Node 'generator_1/latent_vector/Pad' has an _output_shapes attribute inconsistent with the GraphDef for output #0: Dimension 1 in both shapes must be equal, but are 262 and 7. Shapes are [1,262,262,?] and [?,7,7,256].

this code is not working... but the pre-trained dataset works fine.

what should i do? how can i make this checkpoint works fine in this code?
please answer to my question!
Thank You!

Is your encoder fixed or progrresive growing?

Thanks for your work ,it's really interesting.
btw, I wonder is your encoder fixed and linear training, or in a progrresive growing?
thx a lot!

Output looks different at inference time versus at training time

I noticed that when training from scratch the inference output images looks different. I have seen this issue multiple times during development and it is probably due to some misconfiguration in batch renormalization.

Opening this bug to track progress on fixing that.

conditional batch normalization

Hi, could you please tell the difference of instance_norm.py and batch_norm.py?
If I only want to use conditional batch normalization , can I directly call the function conditional_batch_norm() in batch_norm.py to process the feature map.

Why the channel changed after first downsample

From the supplementary of your technical report

Why the channel changed?
Thanks!

ImportError: No module named 'batch_norm' python2????

E:\Users\Raytine\Anaconda3\python.exe F:/zhaiyao/TwinGAN-master/pggan_runner.py
Traceback (most recent call last):
File "F:/zhaiyao/TwinGAN-master/pggan_runner.py", line 39, in
import image_generation
File "F:\zhaiyao\TwinGAN-master\image_generation.py", line 35, in
from libs import ops
File "F:\zhaiyao\TwinGAN-master\libs\ops.py", line 22, in
import libs.instance_norm
File "F:\zhaiyao\TwinGAN-master\libs\instance_norm.py", line 21, in
from batch_norm import get_conditional_batch_norm_param
ImportError: No module named 'batch_norm'

Process finished with exit code 1