How can I add custom attention masks to a Decoder? about x-transformers

[lucidrains]

@DerEchteFeuerpfeil hey Moritz! so i had the same confusion as you starting out in the field with how the mask is represented

so i decided i would not perpetuate this confusion. for all my repos, without exception, masking is always True for attend and False for not. this makes sense to me because if one were to cast it to a float, one can also multiply it with your input for effectively masking in non-attention scenarios

in your example, it should work if you just invert your mask with a ~, but let me know if it does not

[DerEchteFeuerpfeil]

Hey @lucidrains , thanks for the quick reply!

Looks like I was so close to getting it right 😉 This implementation also makes the most sense to me.
Works like a charm now 👍

[lucidrains]

@DerEchteFeuerpfeil nice! go train something amazing :)