renmengye / oc-fewshot-public Goto Github PK

Hi!

I'm not able to replicate the performance for the pretrained online-protonet on the Roaming-Imagenet dataset. With the provided checkpoint I get the reported metrics, but if I pretrain the model I'm getting ~21AP on testing and ~22AP on validation.

To pretrain I use the following command:

python -m fewshot.experiments.pretrain --config configs/models/roaming-imagenet/pretrain.prototxt --env configs/environ/roaming-imagenet-docker.prototxt --seed 0 --tag pretrained

On the pretrain.py file I modified this line: dataset = get_data(env_config) and put this instead: dataset = get_data_fs(env_config, load_train=True), because the method get_data does not exist on the fewshot.experiments.utils file.

Finally, for finetunning/evaluating the pretrained model I run:
python -m fewshot.experiments.oc_fewshot --config configs/models/roaming-imagenet/online-protonet.prototxt --data configs/episodes/roaming-imagenet/roaming-imagenet-150.prototxt --env configs/environ/roaming-imagenet-docker.prototxt --tag the_tag --pretrain results/oc-fewshot/tiered-imagenet/pretrained_model/weights-40000

I hope that you can helps me with this and thanks for the code!

Hi, I've been looking at the code some time and I think i found a bug in the hierarchical episode sampler, specifically on

# Line 169 currently is 
for c in range(min(episode_classes.max(), len(hmap))):
# But should be
for c in range(min(stage.max() + 1, len(hmap))):
# Because each stage is mapped to one of the hierarchy classes

And

# On line 171 it currently is
for c, s in zip(episode_classes, stage):
    # Magic number is 
    results.append(self.hierarchy_dict[hmap[c % len(hmap)]][s])
# The code above chooses the the environment based on the stage relative class number and the 
# class mapped to based on the stage

# But should be reversed to
for c, s in zip(episode_classes, stage):
    # Number of classes of previous stages that belong to the 
    # same class hierarchy as current stage
    mask = ((stage < s) & ((stage % len(hmap)) == (s % len(hmap))))
    prev_samples = np.stack((episode_classes, stage), axis=1)[mask]
    offset = np.unique(prev_samples, axis=0).shape[0]

    # Choose the hierarchy based on the stage and the class on the stage relative class number
    results.append(self.hierarchy_dict[hmap[s % len(hmap)]][c + offset])

Hope I'm not missing something and I'm currently training the models with these modifications, i'll post the results when it get them!