Why is 1GB file can destroy my 12GB colab RAM? about nats-bench

[D-X-Y]

That is because the tss file is compressed when I store it. So that it will take less disk memory, but when you load it, it will decompress and cost more memory.
I would recommend you use archive (tss) with create(None, 'sss', fast_mode=True). This allows you load data on the fly, which means it will only load the data of a specific architecture when you query it.

[huynhnhathao]

I have tried what you said, however when I loop over the whole dataset it takes too long. For example, my function below would take almost 2 hours on google colab to run when fast_mode = True. Is that normal or it's just me? I really expect it to only takes minutes to loop over the whole dataset...

def extract_data(api, dataset):
    # extract flops and test acc for dataset for all architectures
    data = np.zeros(shape = (len(api), 2))
    for i in trange(len(api)):
        flops = api.get_cost_info(i, dataset)['flops']
        test_error = 100 - api.get_more_info(i, dataset, hp = 200)['test-accuracy']
        data[i, 0], data[i, 1] = test_error, flops
    return data

By the way, did you mean create(None, 'tss', fast_mode=True) when using the archive tss?

[D-X-Y]

Yes,, if you traverse all candidates, then the memory cost will be the same and due to the online nature, it will also be slower..
You may need a better machine to run the function. I tried on my personal mac for these functions, which are be completed by < 10 mins

[huynhnhathao]

Ok, thank you for answering my questions.