Comments (10)
Just an update.
We have tried to tweak the architecture (a lot!) and we have double checked the code many times without finding any remarkable problem. Finally, we have turned off the eager execution and we have run the same exact code on the classic TensorFlow workflow. And it worked. Perfectly.
Even if we are not completely sure about this (yet), it really seems like it is an eager execution problem.
We have already reported all here so, if you are interested, feel free to check out the post.
Thank you for your help!
from ganomaly.
Hi, I'm glad that you're interested in this work. Here is my answer to your questions:
1. Inside the update_netd function (in model.py) you have separately called the backward() function first on self.err_d_real and then on self.err_d_fake.
Why instead don't you have applied the backward() pass on self.err_d = self.err_d_real + self.err_d_fake?
Why is there the above mentioned self.err_d = self.err_d_real + self.err_d_fake if it is not used
This part of the code has been adopted from the original DCGAN repository. The generator trains real and fake images separately which is why backward() function is called twice.
2. Why don't you have used the feature matching loss as stated in the paper
I actually did use feature matching loss, but forgot to update the published repository, which is on my to-do list for some time :)
3. What is the meaning of using here the retain_graph=True as the argument of the backward() function? I have tried with retain_graph=False and everything seems works fine as usual
For the generator, we have two sub-networks (auto-encoder and encoder) whose gradients to be computed. After calculating the backward for the autoencoder, PyTorch released the gradients for the autoencoder. When combining three losses (adv + rec + enc), PyTorch v0.3 gives a problem. I'm not sure if this is the case for v0.4. If you don't get any errors, then it might be that this is not a problem anymore.
4. Is the reinitialization of the Discriminator network mandatory? Have you some more comments, instructions, past experience or something else on this particular practice?
For CIFAR and MNIST case, it is not necessary. For some contexts, however, the discriminator gets really good at classifying real vs fake, while the generator still struggles to generate realistic samples. In this case, it is a common practice to re-inialize the discriminator.
Let me know if you have further questions.
from ganomaly.
Thank you for your reply. I have one more question, actually.
- I've implemented the same weight initialization as you do in your code
- I've added the re-initialization of the weights
With this exact configuration, I get from the start that the generator collapses. It shows, since the very beginning, completely random images (chessboard/ QR code like images).
So, I did some tests, and I have proceeded as follows:
- I've completely deleted every batch normalization (from the generator, encoder, and discriminator), the weight initialization and the resetting of the discriminator weights. Result: this gives me good results on generator.
- I've added the batch normalization into generator and encoder (not in the discriminator). Result: I still get good results even if in the long run
- I've added the batch norm even in the discriminator. Result: here, the discriminator loss goes to zero very fast and generator collapses
- I've added the weight reinitialization Result: now the generator seems not to collapse as described in the previous point but the discriminator loss tends to go very often to zero (I will probably leave the model training for a while and see what happens)
Note that now I left the weight initialization as the default Conv layer Keras initialization, i.e.: as a golorot uniform. With this, I get little better results than using the normal initialization as in your code (file: networks.py
, function: weights_init(mod)
)
I don't understand why with the exact same structure I get the generator to collapse instantly whilst with your code I could get very good results from the very "first step". If you have any (and I repeat, ANY) idea it would be great.
Thank you.
from ganomaly.
Thank you for your contribution. The reason for the drop in the accuracy is the overfitting. Once you spot a decrease in the performance, there is no need to train it anymore.
-
Thanks for spotting this.
-
I'll update that part. During the experimentation process, we tried several versions of these. Hence, I sometimes make a mistake to upload the wrong version. Sorry about that.
-
We do use the DCGAN discriminator. The encoder is similar to the discriminator. The difference of the encoder is the dimension of the output channel.
I appreciate these feedbacks. We will be writing a journal extension in which we will be explaining every single bit in more detail.
from ganomaly.
Thanks very much for your information, I am also trying to implement this architecture in Keras. I also realized that there is some difference between the code and the manuscript. Hope I can get a good result.
from ganomaly.
@caiya55 and anyone who's running into the same problem, please check this issue we have opened. Differences in results may be closely related to the "problem" of update ops.
from ganomaly.
thanks a lot for your help. I think this model is very interesting and I am going to introduce this model to solve some classification problem with very imbalanced labels.
After several days' working, I have implemented the model in Keras+ Tensorflow way. The AUC results are similar on MNIST dataset. One little thing that confuses me is that The AUC value may achieve a very high value (like 0.92) in only 2 or 3 epoch, but drops to maybe 0.85 after 10 epochs. I don't know if it is a normal situation. (Take the number 2 as an anomaly). Anyway, I think the result is close to the results reported in the manuscript about MNIST. normally I can reach above 0.8 AUC for each number.
There are several confusing things in the code or the manuscript:
- In Model Test of the manuscript, type-1 loss is used, while in the code, it is type-2 loss.
- In the newest version of the code, feature matching loss is applied for discriminator, actually, in the manuscript, feature matching loss is applied for the Generator. And the adv loss of the Generator in the code use traditional cross entropy loss.
- In the manuscript, the discriminator of DCGAN is applied, while in the code, Discriminator is very similar to the Encoder.
Anyway, I provide a simple version of tensorflow implement. Extra work is required to fine-tuning and visualization.
Best regards.
This is a simple version of Tensorflow implement
from ganomaly.
Thank you for your reply. There is one problem that I encountered now, In MNIST dataset, It seems that the number 1 have very low AUC value (results of other numbers are similar according to your manuscript). I tried different strategies in my Tensorflow version, and I also tried your code in Pytorch. Generally the AUC of number 1 is about 0.3 or 0.4. Did you face this problem before? What is the strategy you use to make it achieve about 0.65 according to your manuscript?
from ganomaly.
@caiya55 and anyone who's running into the same problem, please check this issue we have opened. Differences in results may be closely related to the "problem" of update ops.
hello,can you run the code with tensorflow?
from ganomaly.
I'm closing this issue due to inactivity. Those facing low performance on PyTorch implementation could raise their issue in #44
from ganomaly.
Related Issues (20)
- how can i balance the model HOT 7
- how can i see the result of test?? HOT 1
- How to distinguish a mnist_pic is normal or anomaly? HOT 17
- Sudden increase of err_d HOT 1
- Changes for Windows10
- Some qutestions about testing
- can it display where is the anomaly when inference? HOT 1
- what is the difference between fixed_fakes_*.png and fakes.png
- Is the testing function missing self.netg.eval? HOT 5
- AttributeError: module 'torch._C' has no attribute '_cuda_setDevice' HOT 4
- What's the performance of the model in terms of custom dataset?
- Can this code be used on sequence data? HOT 1
- How to use ganomaly neural network for single-channel data training
- Program stucks when the input image size is 224x224 HOT 2
- Experiment(To replicate the results in the paper for MNIST and CIFAR10 datasets)
- Testing
- Possible mode collapse HOT 1
- Test problem HOT 2
- Missing net.eval() in the testing code HOT 1
- ROC is changed every time
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ganomaly.