neubig / kylm Goto Github PK

Hi,
For sentences made up of mostly unseen tokens, shouldn't the perplexity be high, along with the coverage being low?
This is what I get :
model.arpa : entropy=6.2928944, perplexity=78.40612000159558
model.arpa coverage: 1-gram 11.11111111111111% 2-gram 0.0% 3-gram 0.0%
Even though the coverage is really low(as the tokens are new), the perplexity is also low.
I've also noticed that for sentences which have lesser 'newer' tokens, its the other way round, i.e the perplexity is high

Am I missing something here?
I used the default smoothing option.

Thanks.

What is the technique/algorithm behind in smoothing unigrams to make a model that handles unknown words?