About discriminator, generator and implementation in Tensorflow eager mode about ganomaly

[iLeW]

Just an update.
We have tried to tweak the architecture (a lot!) and we have double checked the code many times without finding any remarkable problem. Finally, we have turned off the eager execution and we have run the same exact code on the classic TensorFlow workflow. And it worked. Perfectly.

Even if we are not completely sure about this (yet), it really seems like it is an eager execution problem.

We have already reported all here so, if you are interested, feel free to check out the post.

Thank you for your help!

[samet-akcay]

Hi, I'm glad that you're interested in this work. Here is my answer to your questions:

1. Inside the update_netd function (in model.py) you have separately called the backward() function first on self.err_d_real and then on self.err_d_fake.

Why instead don't you have applied the backward() pass on self.err_d = self.err_d_real + self.err_d_fake?

Why is there the above mentioned self.err_d = self.err_d_real + self.err_d_fake if it is not used

This part of the code has been adopted from the original DCGAN repository. The generator trains real and fake images separately which is why backward() function is called twice.

2. Why don't you have used the feature matching loss as stated in the paper

I actually did use feature matching loss, but forgot to update the published repository, which is on my to-do list for some time :)

3. What is the meaning of using here the retain_graph=True as the argument of the backward() function? I have tried with retain_graph=False and everything seems works fine as usual

For the generator, we have two sub-networks (auto-encoder and encoder) whose gradients to be computed. After calculating the backward for the autoencoder, PyTorch released the gradients for the autoencoder. When combining three losses (adv + rec + enc), PyTorch v0.3 gives a problem. I'm not sure if this is the case for v0.4. If you don't get any errors, then it might be that this is not a problem anymore.

4. Is the reinitialization of the Discriminator network mandatory? Have you some more comments, instructions, past experience or something else on this particular practice?

For CIFAR and MNIST case, it is not necessary. For some contexts, however, the discriminator gets really good at classifying real vs fake, while the generator still struggles to generate realistic samples. In this case, it is a common practice to re-inialize the discriminator.

Let me know if you have further questions.

[iLeW]

Thank you for your reply. I have one more question, actually.

• I've implemented the same weight initialization as you do in your code
• I've added the re-initialization of the weights

With this exact configuration, I get from the start that the generator collapses. It shows, since the very beginning, completely random images (chessboard/ QR code like images).
So, I did some tests, and I have proceeded as follows:

1. I've completely deleted every batch normalization (from the generator, encoder, and discriminator), the weight initialization and the resetting of the discriminator weights. Result: this gives me good results on generator.
2. I've added the batch normalization into generator and encoder (not in the discriminator). Result: I still get good results even if in the long run
3. I've added the batch norm even in the discriminator. Result: here, the discriminator loss goes to zero very fast and generator collapses
4. I've added the weight reinitialization Result: now the generator seems not to collapse as described in the previous point but the discriminator loss tends to go very often to zero (I will probably leave the model training for a while and see what happens)

Note that now I left the weight initialization as the default Conv layer Keras initialization, i.e.: as a golorot uniform. With this, I get little better results than using the normal initialization as in your code (file: networks.py, function: weights_init(mod))
I don't understand why with the exact same structure I get the generator to collapse instantly whilst with your code I could get very good results from the very "first step". If you have any (and I repeat, ANY) idea it would be great.

Thank you.

[samet-akcay]

Thank you for your contribution. The reason for the drop in the accuracy is the overfitting. Once you spot a decrease in the performance, there is no need to train it anymore.

1. Thanks for spotting this.

2. I'll update that part. During the experimentation process, we tried several versions of these. Hence, I sometimes make a mistake to upload the wrong version. Sorry about that.

3. We do use the DCGAN discriminator. The encoder is similar to the discriminator. The difference of the encoder is the dimension of the output channel.

I appreciate these feedbacks. We will be writing a journal extension in which we will be explaining every single bit in more detail.

[caiya55]

Thanks very much for your information, I am also trying to implement this architecture in Keras. I also realized that there is some difference between the code and the manuscript. Hope I can get a good result.

[iLeW]

@caiya55 and anyone who's running into the same problem, please check this issue we have opened. Differences in results may be closely related to the "problem" of update ops.

[caiya55]

thanks a lot for your help. I think this model is very interesting and I am going to introduce this model to solve some classification problem with very imbalanced labels.
After several days' working, I have implemented the model in Keras+ Tensorflow way. The AUC results are similar on MNIST dataset. One little thing that confuses me is that The AUC value may achieve a very high value (like 0.92) in only 2 or 3 epoch, but drops to maybe 0.85 after 10 epochs. I don't know if it is a normal situation. (Take the number 2 as an anomaly). Anyway, I think the result is close to the results reported in the manuscript about MNIST. normally I can reach above 0.8 AUC for each number.

There are several confusing things in the code or the manuscript:

1. In Model Test of the manuscript, type-1 loss is used, while in the code, it is type-2 loss.
2. In the newest version of the code, feature matching loss is applied for discriminator, actually, in the manuscript, feature matching loss is applied for the Generator. And the adv loss of the Generator in the code use traditional cross entropy loss.
3. In the manuscript, the discriminator of DCGAN is applied, while in the code, Discriminator is very similar to the Encoder.

Anyway, I provide a simple version of tensorflow implement. Extra work is required to fine-tuning and visualization.

Best regards.

This is a simple version of Tensorflow implement

[caiya55]

Thank you for your reply. There is one problem that I encountered now, In MNIST dataset, It seems that the number 1 have very low AUC value (results of other numbers are similar according to your manuscript). I tried different strategies in my Tensorflow version, and I also tried your code in Pytorch. Generally the AUC of number 1 is about 0.3 or 0.4. Did you face this problem before? What is the strategy you use to make it achieve about 0.65 according to your manuscript?

[lzzlxxlsz]

@caiya55 and anyone who's running into the same problem, please check this issue we have opened. Differences in results may be closely related to the "problem" of update ops.

hello，can you run the code with tensorflow?

[samet-akcay]

I'm closing this issue due to inactivity. Those facing low performance on PyTorch implementation could raise their issue in #44