Comments (5)
There may be a negative impact, but it is thought to be small.
To remove the effect, the attention corresponding to the padding token must be set to 0.
You got it right.
from compm.
This link may be helpful.
https://github.com/huggingface/transformers/blob/v4.19.3/src/transformers/models/roberta/modeling_roberta.py#L807
In RoBERTa, if attention_mask is None, the attention of all tokens is 1 by default.
from compm.
If the attention mask for all tokens is 1, wouldn't there be a problem dealing with multiple sequences? (Since the padding input_ids won't be ignored when calculating attention scores)
from compm.
Let me explain with an example.
if batch = 2
sample instance1: [u1; u2; u3]
- predict u3's emotions
sample instance2: [u1; u2; u3; u4]
- predict u4's emotions
input
- Batch input is padded by the length difference between instance1 and instance2.
As you were concerned, We do not set the attention mask for pad tokens to 1.
So it seems that attention mask will be set to 1 even for padding tokens when batch_size is greater than 1.
We missed this part because we set batch_size to 1 when training.
When we train the model, the batch_size is set to 1.
Therefore, these problems did not occur.
Also if batch_size is greater than 1, even if the mask of the pad token is set to 1, it is expected that the model can ignore the padding part while training.
However, your comments will make the model train more effectively.
Thanks.
from compm.
Thank you for your detailed explanation. Can I conclude that during training, when I manually set the batch_size to 16 for example, this will not have a negative impact on training due to attention mask issues? Or am I mistaken and did I read your comment wrong?
from compm.
Related Issues (13)
- Possible typo in the paper HOT 1
- Google Drive link not found HOT 3
- A question about comparison experiment, why didn't you compare with CESTa on dailydialog dataset? HOT 3
- "MELD_models/roberta-large/pretrained/no_freeze/emotion/1.0/model.bin" . It doesn't have mobel.bin file. HOT 2
- reproduce your work on MELD dataset, the weight F1 score is 66.52 HOT 2
- UnboundLocalError: local variable 'DATA_loader' referenced before assignment HOT 1
- RoBERTa large max input token size issue HOT 2
- Unexpected key(s) in state_dict: "context_model.embeddings.position_ids", "speaker_model.embeddings.position_ids" HOT 3
- Can I use this work for sentiment analysis of dyadic conversations? HOT 1
- pytorch-cuda version for the source code? HOT 2
- Can I ignore this warning? UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. HOT 1
- Why don't put batch_speaker_token on cuda? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from compm.