gt-vision-lab / vqa Goto Github PK

I can get the accuracy of val using datatype val2014, through this file : https://github.com/VT-vision-lab/VQA/blob/master/PythonEvaluationTools/vqaEvalDemo.py.
But i have no idea how to get the test-dev accuracy on the papers, because there is no annotation file for test-dev on this page(http://www.visualqa.org/download.html).
The annotation files i can get are list as follows:
mscoco_train2014_annotations.json
mscoco_val2014_annotations.json

No annotation file for test-dev2015.
Any clue?

How was v2_mscoco_train2014_annotations.json generated? I hope to hear from you.

Hi,@AishwaryaAgrawal
I want to get accuracy for test2015, but the Challenge page told me the challenge is ended. But it worked yesterday.
Is there any other ways for me to get the accuracy?

from vpa import VQA, but where is vqa? or How can I install vqa package?

Hi @deshraj @dexter1691 @jiasenlu @StanislawAntol @jwyang @AishwaryaAgrawal,

The Fake Results JSON file provided has only 248349 entries for the train2014 images, while the total number of questions for train2014 images is 443757. Hence, an error is coming up while calculating the accuracy on the Fake Results JSON file. Please do check it.

Thanks.

In vqaEval.py, lines 97-104, the code that computes the accuracy for a generated answer seems to produce strange values. For example, if a question has 8 "yes" answers and 2 "no" answers (provided by the workers), the accuracy of a generated answer would be 0.533 for "no" and 0.2 for "yes".

• accuracy for "yes": 2/10 * min(1, 8/3)
• accuracy for "no": 8/10 * min(1, 2/3)

Can you please explain the reasons for that specific evaluation scheme?

While testing my solution I noticed this odd behavior. (See the picture below)

As you can see, my generated answer is 'none'.
According to the evaluation metric the correct accuracy should be 30% because there is one answer the same as mine.
I think this is happening because of the processing done before evaluation. In file vqaEval.py line 42, the answer 'none' is replaced with '0' . Because there is not any '0' in ground truth answers, the accuracy is set to 0.00%. If I remove 'none': '0', from manualMap dictionary I get the right accuracy for this question (30%).

If it helps, the id of this question is 411188011, and the name of the picture is COCO_val2014_000000411188.jpg

Can you look more into it? I hope I didn't miss anything.

Excuse me, there are some files can not download,


Including ./Annotations ./Questions

Hello, I am building a VQA system and am seeking clarification for a condition in the vqaEval.py script.

if len(set(gtAnswers)) > 1:
    for ansDic in gts[quesId]['answers']:
        ansDic['answer'] = self.processPunctuation(ansDic['answer'])
        ansDic['answer'] = self.processDigitArticle(ansDic['answer'])
    resAns = self.processPunctuation(resAns)
    resAns = self.processDigitArticle(resAns)

The above condition is placed before running standard post-processing on predictions in line 98. My understanding is that this translates to: 'if all the human annotators agree, don't do any post-processing. However, my system is producing some variation in outputs in these cases, such as 'yes!' rather than 'yes'. Of course I can do my own post-processing, but I was wondering if you might offer some insight into the rationale behind the above condition?

Many thanks!

When I access the website, I see no links for test answer files.

from external.vqa.vqa import VQA
import os
data_dir = "../../../data/vqa/data"
annotation_json_file_path = os.path.join(data_dir, "mscoco_train2014_annotations.json")
question_json_file_path = os.path.join(data_dir, "OpenEnded_mscoco_train2014_questions.json")

vqa = VQA(annotation_json_file_path, question_json_file_path)

q_requested = vqa.loadQA(ids=[409380])
img_id_requested = vqa.getImgIds(quesIds=[q_requested[0]['question_id']])

output:
Traceback (most recent call last): File "q1.py", line 13, in <module> img_id_requested = vqa.getImgIds(quesIds=[q_requested[0]['question_id']]) File "external\vqa\vqa.py", line 113, in getImgIds anns = sum([self.qa[quesId] for quesId in quesIds if quesId in self.qa], []) TypeError: can only concatenate list (not "dict") to list

is this the expected behaviour? I modifed line 113 in vqa.py to return the image_id as below but not sure if it will break something else.

anns = [self.qa[quesId] for quesId in quesIds if quesId in self.qa]