Training hours & Imagenet accuracy about nested-transformer

[zizhaozhang]

Hi is that possible your actual data disk is away from you machine, so the latency is due to data pipeline.

We have trained tiny version for a few times, it seems there exist some variance around 0.3%. Your l2_grads seems much higher than mine, which is suspicious to me. can you look into from this point?

[arunos728]

Thanks for the quick answer. I'll check your point about l2_grads.
One more thing, did you use TPU pod slices (ex. 8x4 or 8x8) when training the NesT-tiny? or just used a single TPU?

[zizhaozhang]

I am using 2x2 TPU for NesT-tiny (I think it is called single TPU).