Target tokens for loss computation about maskgit-pytorch

[llvictorll]

Hi,

Yes, there is a difference between my repo and the one from lucidrains. I do not know if the official repository use the same strategy since they do not release the training script.

1. I think that keeping the prediction on the 'known (unmasked)' token will help to quickly see if the network is learning or not, for debugging purposes. I do not think that the prediction will be biased towards the 'known (unmasked)' tokens because the masking ratio is high (i.e. arccos scheduler) and the loss for 'known (unmasked)' tokens will be very low very quickly.

2. Here is the learning curve from the shared model. It shows some picking in the gradient curve (epoch 8, 33, and 85) that might be eliminated by more drastic gradient_norm_clipping.

[yukara-ikemiya]

Thank you for the respose. The loss curve plot is very helpful for me.