How long does it take to complete the training run? (hint: this session is on distributed training, so it will take a while)
It took 1h 53min to complete te training run
No. The model checkpoint is generated every 4k interactions.
No. If we were overfitting, we would see that the accuracy of our model on the validation data would peak after training for a number of epochs, and would then start decreasing
Yes. When running the nvidia-smi command, we can see that the GPUs are at 100% Util (see image attached here named CPU.jpeg)
Yes, I monitored. It was not a bottleneck.
Take a look at the plot of the learning rate and then check the config file. Can you explan this setting?
The learning rate decreases with the incrementality of the BLEU score.
To train we used the file: train.clean.en.shuffled.BPE_common.32K.tok
It has 958.6MB
It has 4524868 lines
val_loss=1.5943-step-44012.data-00000-of-00001 . -- Model
val_loss=1.5943-step-44012.index . --- Index
val_loss=1.5943-step-44012.meta . --- Meta
it is 852.3MB
In tensorflow one steps is considered as number of epochs multiplied by examples divided by batch size.
The average step took 0.59 seconds
The step duration is directly proporcional to the network utilization between nodes.
The higher the utilization, higher will be the time to run the step.