Should we specify attention_mask when using gpt2 for the conversation task? about crslab

[txy77]

Thanks for your feedback. @Oran-Ac

1. The padding labels are set to -100 according to your suggestion.
2. Due to the limitation of gpu memory, we cuf off the context when doing the generation.
3. The position_ids will be automatically added by Transformers if no parameter is passed.

[Oran-Ac]

I think when doing the batch generation, we also need to pass the position ids to the model. Otherwise, the position ids will always be 1 in the loop except for the first round. You can find the discussion [here].(huggingface/transformers#3021 (comment))

CRSLab/crslab/model/conversation/gpt2/gpt2.py

Line 98 in b3ab262

outputs = self.model(context, former_hidden_state) # (bs, c_t, v_s),

And why you cut off the context when doing the generation?

CRSLab/crslab/model/conversation/gpt2/gpt2.py

Line 95 in b3ab262

context = context[..., -self.response_truncate + 1:]