Mask-Filling with pretrained BORT about bort

[adewynter]

Hi!

That's definitely a very odd error. Have you tried using the mask token implicitly? E.g., like in here. The reason being is that this particular version of Gluon is based on a script that did not originally support RoBERTa-style pretraining, so we kind of had to hack our way around that.

I'm unsure if the actual mask token in Gluon+RoBERTa + this version is defined as "<MASK>", so perhaps you could change it to vocab.mask_token and try again? Indeed, note how the output token is actually lowercase ("<unk>") on the notebook you shared.

Sorry about that!

-Adrian

PS: We never did mask prediction because Bort was not designed for that -- indeed, we have conjectured in the paper that MLM/pre-training is not something that is even needed. So, you should get "a" token (you shouldn't get the unknown token for sure!), but I'm not superduper sure if it'll be a sensible token, as you say.

[adewynter]

Hey hey, did it work? I checked on this page and looks like lowercasing the tokens was the way to go. Should I close this then?

[patrickvonplaten]

Sadly, @stefan-it and I didn't get it to work yet :-/. As you can see on this page the proposed word to fill the mask don't seem to be sensible. I tried to make it work using gluonnlp and mxnet in the notebook posted above without success. If you find a bit of time to make the mask-filling work in the gluonnlp library this would be super helpful!