can the pre-trained model be used as a language model? about bert

[jacobdevlin-google]

It can't, you can only use it to get probabilities if a single missing word in a sentence (or a small number of missing words). This is one of the fundamental ideas, that masked LMs give you deep bidirectionality, but you no longer have a well-formed probability distribution over the sentence (which in general, we don't care about).

[xu-song]

What about mask each word sequentially. Then score sentence by summary of word score.

[Bachstelze]

Alex Wang and Kyunghyun Cho are using in BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model the unnormalized log-probabilities to rank a set of sentences. For this purpose it seems to work.

[WolfNiu]

What about mask each word sequentially. Then score sentence by summary of word score.

That way your calculation won't be correct.

Let's say the sentence has only two tokens x1 and x2. Your calculation well give P(x1 | x2) * P(x2 | x1), which doesn't lead to the probability of the whole sentence. Note that this is not to say what you intended was not doable -- it's just that your way probably won't work.

[Shujian2015]

You can fine-tune BERT to be LM: https://arxiv.org/abs/1904.09408

[hscspring]

using BERT as a language Model · Issue #37 · huggingface/pytorch-pretrained-BERT

It's actually like what @jacobdevlin-google have said, bert is really not a language model.