Lower than expected final test accuracies for some models? about nasbench

[wendli01]

As far as I understand it, this happens sometimes with tensorflow when the learning rate is borderline too high (for the selected model).
It can then be a result of the model constantly overshooting local optima because of the high step size and some tensorflow internal safeguards that basically output predictions as a dummy classifier.

Usually this behavior is also quite stochastic, meaning it might only appear in one of the 3 runs.
For this reason, some work simply disregards architectures that achieve <80% final accuracy as noise.

However, it appears as if the models you listed could not be trained in all 3 runs, so across 3 different random inits.
Also the models were trained with cosine lr decay which should prevent this kind of training problem in the first place.

It is probably an issue with the TPU v2 architecture.