gt-vision-lab / vqa Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
I can get the accuracy of val using datatype val2014, through this file : https://github.com/VT-vision-lab/VQA/blob/master/PythonEvaluationTools/vqaEvalDemo.py.
But i have no idea how to get the test-dev accuracy on the papers, because there is no annotation file for test-dev on this page(http://www.visualqa.org/download.html).
The annotation files i can get are list as follows:
mscoco_train2014_annotations.json
mscoco_val2014_annotations.json
No annotation file for test-dev2015.
Any clue?
How was v2_mscoco_train2014_annotations.json generated? I hope to hear from you.
Hi,@AishwaryaAgrawal
I want to get accuracy for test2015, but the Challenge page told me the challenge is ended. But it worked yesterday.
Is there any other ways for me to get the accuracy?
from vpa import VQA, but where is vqa? or How can I install vqa package?
Hi @deshraj @dexter1691 @jiasenlu @StanislawAntol @jwyang @AishwaryaAgrawal,
The Fake Results JSON file provided has only 248349 entries for the train2014 images, while the total number of questions for train2014 images is 443757. Hence, an error is coming up while calculating the accuracy on the Fake Results JSON file. Please do check it.
Thanks.
In vqaEval.py
, lines 97-104, the code that computes the accuracy for a generated answer seems to produce strange values. For example, if a question has 8 "yes" answers and 2 "no" answers (provided by the workers), the accuracy of a generated answer would be 0.533 for "no" and 0.2 for "yes".
2/10 * min(1, 8/3)
8/10 * min(1, 2/3)
Can you please explain the reasons for that specific evaluation scheme?
While testing my solution I noticed this odd behavior. (See the picture below)
As you can see, my generated answer is 'none'
.
According to the evaluation metric the correct accuracy should be 30% because there is one answer the same as mine.
I think this is happening because of the processing done before evaluation. In file vqaEval.py
line 42
, the answer 'none'
is replaced with '0'
. Because there is not any '0'
in ground truth answers, the accuracy is set to 0.00%. If I remove 'none': '0',
from manualMap
dictionary I get the right accuracy for this question (30%).
If it helps, the id of this question is 411188011
, and the name of the picture is COCO_val2014_000000411188.jpg
Can you look more into it? I hope I didn't miss anything.
Hello, I am building a VQA system and am seeking clarification for a condition in the vqaEval.py script.
if len(set(gtAnswers)) > 1:
for ansDic in gts[quesId]['answers']:
ansDic['answer'] = self.processPunctuation(ansDic['answer'])
ansDic['answer'] = self.processDigitArticle(ansDic['answer'])
resAns = self.processPunctuation(resAns)
resAns = self.processDigitArticle(resAns)
The above condition is placed before running standard post-processing on predictions in line 98. My understanding is that this translates to: 'if all the human annotators agree, don't do any post-processing. However, my system is producing some variation in outputs in these cases, such as 'yes!' rather than 'yes'. Of course I can do my own post-processing, but I was wondering if you might offer some insight into the rationale behind the above condition?
Many thanks!
When I access the website, I see no links for test answer files.
from external.vqa.vqa import VQA
import os
data_dir = "../../../data/vqa/data"
annotation_json_file_path = os.path.join(data_dir, "mscoco_train2014_annotations.json")
question_json_file_path = os.path.join(data_dir, "OpenEnded_mscoco_train2014_questions.json")
vqa = VQA(annotation_json_file_path, question_json_file_path)
q_requested = vqa.loadQA(ids=[409380])
img_id_requested = vqa.getImgIds(quesIds=[q_requested[0]['question_id']])
output:
Traceback (most recent call last): File "q1.py", line 13, in <module> img_id_requested = vqa.getImgIds(quesIds=[q_requested[0]['question_id']]) File "external\vqa\vqa.py", line 113, in getImgIds anns = sum([self.qa[quesId] for quesId in quesIds if quesId in self.qa], []) TypeError: can only concatenate list (not "dict") to list
is this the expected behaviour? I modifed line 113 in vqa.py to return the image_id as below but not sure if it will break something else.
anns = [self.qa[quesId] for quesId in quesIds if quesId in self.qa]
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.