Key Error When reshaping position embedding. about albef

[LiJunnan1992]

Hi, the coco (fine-tuned) checkpoint does not need to reshape the positional embedding, nor does it have visual_encoder_m. The checkpoint loading script in Retrieval.py is intended to load from a pre-trained model. You can remove the reshape code if you want to load from a fine-tuned checkpoint.

Thanks!

[Ngheissari]

Hi, the coco (fine-tuned) checkpoint does not need to reshape the positional embedding, nor does it have visual_encoder_m. The checkpoint loading script in Retrieval.py is intended to load from a pre-trained model. You can remove the reshape code if you want to load from a fine-tuned checkpoint.

Thanks!

Hi, trying to finetun on refcoco using grounding from pre trained checkpoint (ALBEF.pth ) as a sanity check and it complains about :
m_pos_embed_reshaped = interpolate_pos_embed(state_dict['visual_encoder_m.pos_embed'],model.visual_encoder_m)

saying object ALBEF has no visual_encoder_m. If I replace model.visual_encoder_m with visual_encoder it works. So is it a problem with the code or checkpoint ?

Thank you for your work.