[BUG] Trained PPO from scratch won't work with Analyzer about convlab-2

[zqwerty]

Thank you!
As the warning:

WARNING:transformers.tokenization_utils_base:Token indices sequence length is longer than the specified maximum sequence length for this model (975 > 512). Running this sequence through the model will result in indexing errors

I agree with you that the sequence is too long for jointBERT model. I will cut too long sentences in jointBERT.

And yes, this is the right approach for training and testing a RL policy from scratch. But as you can see, the system response is too long because your policy outputs too many dialogue acts, which indicates the policy is not well trained.

[thenickben]

Thank you! I didn't see the warning, it seems it doesn't print on console on Colab.

Another thing I've noticed is that if I train for more epochs (e.g. 1000) the policy loss eventually gets "nan".

Could it be the case that the hyperparameters in the config file aren't the optimal ones? Or are those the same as you've used when reached 0.74 success rate? Cheers!

[zqwerty]

First MLE then PPO: #15 (comment)
Notice: I've improved user agenda policy, so this number may not be accurate, see README https://github.com/thu-coai/ConvLab-2#policy

[zqwerty]

move to #54