hsinger04 / vogue-reimplementation Goto Github PK

1. Have a 2-layer MLP map from w to sigma. Only a single one is needed, as w_latent stays the same for all styles / resolutions.
2. Have p as a trainable vector, q = sigmoid(p) and Q = DiagonalMatrix(q) for each style.
3. Calculate losses and optimize.

Either: Reduce image and segmentation dimensions or change StyleGAN to input / output 1024 images

1. Create a CNN that maps from image to latent vector z.
2. Train it by minimizing the perceptual loss between input and output by StlyeGAN (unclear: perceptual loss)

• Things to optimize – prefer generator code: Simple to write and memory-efficient. And maybe try_on doesn‘t need long to train
  • Speed
  • Memory
• What the dataset should look like
  • Returns: latent_p, latent_g, seg_p, seg_g

1. Project images to latent vector z
2. Learn try-on

Relevant code sections: TryOn.ipynb, generator, modulated_conv2d

Still need to understand it. Very mysterious code.

1. Calculate A (Is the style matrix. See Fig. 2 in Editing in Style.)
2. Get Segmentation (tf.resize)
3. Normalize A
4. (Downsample Segmentation)
5. Follow Eq. 3 from VOGUE (Just multiply A^2 with U and then use reduce_mean)
6. Unclear: Right side of page 4

In particular, what A is.

• Refactor code
• Fix Encoder (find out why it fails by maybe inspecting the loss (via tensorboard) and find an alternative architecture)
• GitHub README and .yaml
• Analysis of results

Ideally, we would use the original authors' dataset. However, I also found https://github.com/royorel/FFHQ-Aging-Dataset, which is good for the following two reasons:

1.) It builds on the FFHQ dataset, which StyleGAN was also trained on --> might allow us to be more flexible with regards to using transfer learning
2.) It contains segmentation labels

Dataset still needs to be split

@return lambda: imsize = 256

In https://github.com/HSinger04/VOGUE-Reimplementation/blob/main/cryu854/modules/layers/modulated_conv2d.py under
s = self.fully_connected(w_latents) + 1

Try ResNet50 instead of VGG. Reason: https://www.researchgate.net/figure/Comparison-of-popular-CNN-architectures-The-vertical-axis-shows-top-1-accuracy-on_fig2_320084139 and https://arxiv.org/pdf/2006.12655.pdf

self.p initialized with random_normal or random_uniform leads to error, even though I am 99% sure I am supposedly doing it correctly.

Suggestions:

• https://papers.nips.cc/paper/2019/file/af8d9c4e238c63fb074b44eb6aed80ae-Paper.pdf
• https://github.com/adambielski/perturbed-seg
• https://openaccess.thecvf.com/content_ICCVW_2019/papers/CVFAD/Korneliusson_Generative_Modelling_of_Semantic_Segmentation_Data_in_the_Fashion_Domain_ICCVW_2019_paper.pdf
• https://link.springer.com/article/10.1007/s13735-017-0141-z
• https://arxiv.org/abs/1707.02051
• https://ieeexplore.ieee.org/abstract/document/8424314
• https://arxiv.org/abs/1704.06857
• https://openaccess.thecvf.com/content_ECCV_2018/papers/Liang-Chieh_Chen_Encoder-Decoder_with_Atrous_ECCV_2018_paper.pdf

• Seems to somehow only map to positive numbers right now.
• Batch Normalization and tanh are missing
• Make sure to save current model files somewhere else and start retraining.

1. Just try out L2-norm between unit-scaled activations of last layer between two networks or sth.

Needs to be done with cryu854