Reproduce the result on okvqa about lavis

[dxli94]

Hi @YHWmz ,

The 55.4 result is obtained by initializing from VQAv2 checkpoint. Can you confirm you are loading this checkpoint?

If directly tuned on OKVQA from pretraiend checkpoint, ~45 is expected.

Thanks.

[YHWmz]

Hi @YHWmz ,

The 55.4 result is obtained by initializing from VQAv2 checkpoint. Can you confirm you are loading this checkpoint?

If directly tuned on OKVQA from pretraiend checkpoint, ~45 is expected.

Thanks.

Thanks for your replay.

I find that the model initializing from 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_vqa_capfilt_large.pth' in LAVIS/lavis/models/base_model.py line 101.

I am not sure whether the ‘model_base_vqa_capfilt_large.pth’ is VQAv2 checkpoint？By the way, I didn't change the code in LAVIS but just run the script LAVIS/run_scripts/blip/train/train_okvqa.sh

[YHWmz]

I also try to run LAVIS/run_scripts/blip/train/train_aokvqa.sh and the model is initializing from 'model_base_vqa_capfilt_large.pth' as well. Then I find the result is ~50, which also lower than 56.2 in Benchmark.

[dxli94]

Are you using the same configurations as in .yaml? Note that OKVQA and AOKVQA dataset sizes are quite small. If you change batch size etc, you may also need to adjust other hparams accordingly to reach the best results.

[YHWmz]

Oh, I change the bz to 12. Maybe that is the problem.

I'll let you know when I figure it out.

Thank u so much for the help~

[YHWmz]

Solved.

The problem is the bz.

Thanks for your help again.

[dxli94]

Glad to hear.