Training loss about electra

[008karan]

You can always use tensorboard though!

[chriskhanhtran]

@008karan Hi Karan. I am not a TF user. Can you please instruct me how to use tensorboard in this case? I see an tfevents file in the model directory but it seems not to be written for tensorboard. The script used tf.estimator.tpu.TPUEstimator to train the model so I don't know how to extract the loss. Thank you very much in advance!

[008karan]

I have trained my model on GPU and using tensorboard is similar here. You will find events.out files in your checkpoint folder. Just run tensorboard on it.

[chriskhanhtran]

@008karan Thank you Karan! It works for me now after I use tensorboard==1.15.0. Do you know how the author can continuously get the evaluation metrics as in this thread? I can only get the evaluation metrics at the end of my training progress.

[008karan]

In tensorboard you get loss and learning rate here I think you can add whatever you want in logs to see them on tensorboard!

[mrm8488]

When I trained Electra small on my Spanish corpus the loss was shown if trained on GPU. Now, I got access to TFRC and trained it using its TPUs pod and loss it is not shown. Of course, I can get it from Tensorboard events but would be great to log it by default when running on TPU.

[nemani]

You can set the tensorflow log level to info and it will be much more verbose including printing the loss.