train_acc1es @200 and info.train-accuracy don't match! about nats-bench

[D-X-Y]

@debadeepta Thanks for the question.
I created a small example at https://github.com/D-X-Y/NATS-Bench/blob/main/nats_bench/api_test.py#L63
But I did find info['train-accuracy'] is 84.038? Would you mind adding the full codes to reproduce this problem?

[D-X-Y]

You can check the args of get_more_info function here: https://github.com/D-X-Y/NATS-Bench/blob/main/nats_bench/api_utils.py#L218
We have a parameter: is_random (default value is True). This parameter controls whether to return a randomly selected single run or use the average.
If you want to use the average, you can use api.get_more_info(284, 'cifar10', hp=200, is_random=False).

[debadeepta]

Actually I made a mistake and I was wrong. The below code blurb prints out the same values as expected.

data = api.query_by_index(284, dataname='cifar10', hp='200')
data[777].train_acc1es[199]
94.568
info = api.get_more_info(284, 'cifar10', hp=200)
info['train-accuracy']
94.568

But now I have a new doubt:
The three different runs of architecture 284 for 200 epochs have the following accuracies:
data[777].train_acc1es[199]
94.568
data[888].train_acc1es[199]
94.772
data[999].train_acc1es[199]
94.728

Shouldn't the value in api.get_more_info be the average of these three runs instead of the result of 777 seed?

[debadeepta]

Ahh nice. Thanks a lot again!