The reasults of test accuracy about nats-bench

[D-X-Y]

cifar10-valid and cifar10 are a bit tricky.
If you try:

info = api.get_more_info(
            1,
            dataset = "cifar10",  # not cifar10-valid
            hp="200" if search_space == "tss" else "90",
            # hp='12',
            is_random=False
        )
print(info)

Then the test accuracy is the same as what you got from api.query_info_str_by_arch.

Basically, when you indicate cifar10-valid, we aim to get the validation accuracy on CIFAR-10, for which the training set is 50% subset of the original CIFAR-10 training set. When you indicate cifar10, we are using the full original CIFAR-10 training set for training.

[tjdeng]

I got it. Thank you for your reply.