Comments (1)
We use decoder_prompts
and prefix_tokens
for better VQA finetuning performance. Specifically, for VQA we have an hyper-parameter option called --prompt-type
, which determines whether to add the question before the answer in the input sequence of the decoder during finetuning & evaluation. The question has already been input in the encoder, here we consider whether to feed it into the decoder again. If the --prompt-type
is not none
, then the decoder_prompts
and prefix_tokens
will record the prepended question to construct the decoder input sequence during evaluation. The decoder_prompts
is used for all-candidate evaluation and the prefix_tokens
is used for beam-search generative evaluation. In our experiments, we found concatenating the question with the answer in the decoder input sequence improves the accuracy somewhat, compared with not performing concatenation.
For the other question, note that in the original VQAv2 dataset, most questions are annotated with more than one ground-truth answers. However, OFA is a seq2seq model which requires one source sequence (image & question) paired with only one target sequence (ground-truth answer) during training. In this case, we split the original sample with one question paired with more than one answers into multiple seq2seq samples, each consists of the question paired with one of the ground-truth answer.
from ofa.
Related Issues (20)
- visual grounding task
- How to perform multi frame detection for visual grounding tasks?
- Inquiry Regarding the Acquisition and Processing Details of text2phone_dict.txt
- How to generate a paragraph instead of one single sentence through image caption model?
- TransformerDecoderLayer.py
- Pi Tuning
- How to calculate expected time reduction rate
- pip install error HOT 1
- How to use OFA to complete the task of text to sql?
- Grounded Captioning
- Question about data processing in pretrain data. HOT 1
- Is the code for the downstream task of detection open source?
- Improve finetune efficiency
- MuE training script
- How to perform text infilling
- How can I train my own data set in an inmage description task?
- Batch to Inference
- The Chinese universal and OCR experience area of OFA does not match the test inference results of this model page
- Building wheels for collected packages: transformers install a wrong version of transformers (4.18.0) which is not compatible with sentence-transformers HOT 1
- OFA Base weight file over 800 M
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ofa.