Documentation for NATS Bench data format about nats-bench

[D-X-Y]

@610v4nn1
First of all, thanks for your interest.

If your project is based on Python, I would suggest you directly clone this repo (https://github.com/D-X-Y/NATS-Bench) and paste it into your project. The NATS-Bench has a minimal dependency: abc, bz2, pickle, numpy, and warning.

If you are interested in the data format, all the files are saved based on bz2/pickle, and you can use this function (https://github.com/D-X-Y/NATS-Bench/blob/main/nats_bench/api_utils.py#L89) it.
The loaded data in Python is usually a dict, where the key indicates the hyperparameters -- 12 or 90 or 200, and the value is all the information of a specific architecture trained with this hyperparameter. You can find more details at https://github.com/D-X-Y/NATS-Bench/blob/main/nats_bench/api_utils.py#L275.

Please let me know if you have any other questions!

[610v4nn1]

Thanks for the answer @D-X-Y

I managed to load and parse the dictionaries before, but I still have some problems interpreting the meaning of the keys.
I need to extract some simple informations:

• value of each hyperparameter (e.g., number of layers, different type of operations)
• value for each metric (e.g., accuracy) both on the validation set and on the test set when available. This also required to know at which epoch of training the measurement was made

I just want to get a simple CSV file with these info out of the complex dictionary format.

[610v4nn1]

Just to give you an idea, my format is pretty much like this

`
id, hp_lr, hp_max_dropout, hp_max_units, ...., metric_validation_f1_micro, metric_validation_accuracy, metric_test_f1_micro

1, 0.1, 0.25, 128, ...., 0.1, 0.4, 0.12
`

[D-X-Y]

If so, a simple way could be to load NATS-Bench on your laptop and save the metrics that you are interested in to a CSV file; and finally, load this CSV in your software.