Confidence of the BLIP captioning model about lavis

[karndeepsingh]

Hi,

Is there a way to access the confidence of the generated caption?

@jhwang7628 I have a similar question. Did you figure out any way to get the confidence level?

[dxli94]

Hi @jhwang7628 @karndeepsingh

Currently this is not natively supported. I have to look into whether this is possible for HF bert. This expects delay.

One alternative might be to forward the generated caption and the original image into the BLIP ITM model to get a matching score.

Thanks.

[karndeepsingh]

Hi @jhwang7628 @karndeepsingh

Currently this is not natively supported. I have to look into whether this is possible for HF bert. This expects delay.

One alternative might be to forward the generated caption and the original image into the BLIP ITM model to get a matching score.

Thanks.

Thanks @dxli94 for answering.
How can I get a confidence score for VQA using BLIP?
Also, How can I compare the results of VQA and report the accuracy of the model on my dataset ( I am only using a pre-trained model for now)?

[karndeepsingh]

Hi @jhwang7628 @karndeepsingh

Currently this is not natively supported. I have to look into whether this is possible for HF bert. This expects delay.

One alternative might be to forward the generated caption and the original image into the BLIP ITM model to get a matching score.

Thanks.

@dxli94 How i can use BLIP ITM on text and image to predict the confidence? Any resource or notebook can you share? Is it possible using lavis ?