Comments (4)
Main issues:
- Synthetic data: generating repeated sentences (extreme mode collapse) when gpre_lr=0.005, while your model behaves normal under gpre_lr=0.01.
Right after pre-training, the nll_gen is also very high (~0.65 as you showed). So I think it’s because by setting gpre_lr=0.005 and npre_epochs=150, the pre-training may not be sufficient. A quick suggestion is to increase the npre_epochs (and/or increase the inverse temperature) to see if you can get good results.
- Image COCO data: generating repeated sentences when temperature=1.
It would be as expected since there is a tradeoff between sample quality and diversity, tuned by the maximum inverse temperature. In the extreme case where temperatue=1, which means no temperature control at all, the model will suffer from severe mode collapse.
- Reasons of generating repeated sentences:
Very sensitive to temperature?
Type of adversarial training loss?
Or others?
For the gumbel-softmax trick, the temperature control plays a crucial role in the overall performance. So yes, it is mainly because “very sensitive to temperature”.
- Could you please explain why your model would arise the problem of generating repeated sentences? Or what modules do you think would lead to this problem?
The gumbel-softmax trick.
- Do you have any suggestions for solving the mode collapse problem?
I would recommend to improve the gumbel-softmax trick. In this work, we just use the vanilla version of the gumbel-softmax with some temperature control. I believe there is still large room for improving this module. For example, REBAR would be the first thing to try.
- Previous works like SeqGAN and LeakGAN don’t have extreme mode collapse (only generate one repeated sentence) even the temperature is set to 1. Do you think the temperature exceed 1 is essential to your RelGAN?
Yes, temperature>1 is essential for RelGAN from the temperature control perspective. SeqGAN and LeakGAN do not reply on temperature control as they apply REINFORCE, so they are less sensitive to the temperature.
from relgan.
I am very grateful that you can take your time to answer my questions in detail and patiently. Your answer does help me have a better understanding of your RelGAN, while I am also confused about another thing.
According to my understanding of your code, the calculation process of g_pretrain_loss
and nll_gen
is exactly the same, except for the parameters of the “generator”. In fact, nll_gen
is the g_pretrain_loss
calculated by the "generator" whose parameters have been updated after pre-training. Therefore, the value of g_pretrain_loss
and nll_gen
should be close, and the value of g_pretrain_loss
should be generally larger than the value of nll_gen
from the training perspective. However, from the log file with gpre_lr=0.005
on Synthetic Data, the g_pretrain_loss
is already small (~1.7) while the nll_gen
is still large (~6.3). According the above analysis, this situation should not happen.
Is there mistake of my understanding or my analysis? Or is the way you calculate the g_pretrain_loss
and the nll_gen
different?
from relgan.
I think the difference between g_pretrain_loss
and nll_gen
mainly lies in how each of them is calculated over mini-batches: For nll_gen
, we fix the generator parameters and take average of g_loss
over all mini-batches (please refer to nll_loss()
in Nll.py
). For g_pretrain_loss
, however, we always first adapt the value of g_loss
to each mini-batch and then take average of the adapted g_loss
s over all mini-batches (please refer to pre_train_epoch()
in utils.py
). It explains why g_pretrain_loss
is lower than nll_gen
.
from relgan.
Thank you again for your answers and code :)
from relgan.
Related Issues (20)
- NLL_gen loss with initialization HOT 14
- Where generated text is being stored? HOT 1
- How much time it took to run a single cycle of 5 discriminator steps and 1 generator step during the adversarial training? HOT 3
- Overfitting during Pretraining HOT 3
- AttributeError: module 'tensorflow' has no attribute 'truncated_normal_initializer' HOT 1
- Teacher Forcing during Adversarial Training HOT 1
- Facing the error HOT 8
- Why Gumbel-Softmax relaxation applied in sample generation? HOT 4
- Dis loss decays to zero while training oracle HOT 3
- How do i train the model with my own data? HOT 2
- Loss function for LSTM based model HOT 1
- LICENSE? HOT 2
- Questions on reproducing the results HOT 18
- How long is expected to take to complete the pre-training stage (on single GPU) ? HOT 4
- Thank you for your contribution. Can you briefly explain how to run and validate your model? HOT 1
- Improving sample output quality RELGAN. HOT 2
- my idea of metric on diversity. HOT 6
- [Self-BLEU] what metric values do you get? HOT 4
- Using RelGAN for paraphrase generation HOT 1
- Closed. HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from relgan.