learningbydoingcompetition / learningbydoing-comp Goto Github PK

When re-generating competition data locally, please note that some hash IDs might not match between competition data and locally generated data. The exact cause is unknown, but possibly due to configuration/software differences between when we originally generated the competition data several months ago and now, when we open-sourced our code.

Importantly, all trajectories are completely identical to what was used in the competition -- it's just the hash ID that might not match for some files. We've attached a screenshot of a visual diff (using Meld) to illustrate what we mean -- see more info below.

Among other uses, the hash ID was used to ensure consistency between evaluating systems on Codalab as a security measure.

If you want to re-generate competition data, we recommend first re-generating the training and testing data using the scripts in our repository, then compare these files with the corresponding competition datafiles under data/. Most files will be identical, but a few will only differ because of hash ID mismatches. You can copy these files over from the competition data to keep the same hash IDs that were used in the competition.

We would love to pinpoint the cause, but we looked into this issue a bit but couldn't find a workaround (mostly by trying to set PYTHONHASHSEED to something that didn't yield hash ID mismatches). So, we're posting an issue to notify others. In case you re-generate the data and don't see this issue -- please let us know! We'd love more information to help track this issue down and fix it!

Detailed info:

Summary of hash ID mismatches:

• CHEM train: 20 of 240 datafiles showed a hash ID mismatch
• ROBO train: 4 of 1200 datafiles showed a hash ID mismatch
• ROBO test: No hash ID mismatches! (0 of 960 datafiles)

CHEM train datafiles with a hash ID mismatch:

system_03_instance_00.csv
system_03_instance_04.csv
system_03_instance_16.csv
system_04_instance_02.csv
system_04_instance_06.csv
system_04_instance_07.csv
system_04_instance_08.csv
system_04_instance_11.csv
system_04_instance_16.csv
system_04_instance_17.csv
system_04_instance_18.csv
system_07_instance_08.csv
system_07_instance_11.csv
system_07_instance_12.csv
system_07_instance_17.csv
system_07_instance_19.csv
system_08_instance_02.csv
system_08_instance_04.csv
system_08_instance_07.csv
system_08_instance_09.csv
system_08_instance_14.csv
system_08_instance_16.csv
system_08_instance_17.csv
system_09_instance_04.csv
system_09_instance_08.csv
system_10_instance_02.csv
system_10_instance_05.csv
system_10_instance_09.csv
system_10_instance_13.csv
system_10_instance_19.csv

ROBO train datafiles with a hash ID mismatch:

rebel-mauve-beetle_09.csv
talented-steel-butterfly_21.csv
talented-steel-butterfly_38.csv
thoughtful-steel-butterfly_38.csv

Screenshot of visual diff for a CHEM train datafile (system_03_instance_00.csv) showing that only the hash ID differs, but the entire trajectory is identical (this file is representative of all datafiles with a mismatching hash ID):