Weird classification for Latvian & ex-Yougoslavian languages about langid.py

[saffsd]

Hi Pierre,

Thanks for getting in touch! Misclassifying Latvian to Luxemburgish does indeed seem strange. I'm not familiar with either language but it's not a confusion pair I have heard of before, so it seems quite plausible that there may be some incorrectly labeled training data. Could you share some examples of incorrectly-labeled sentences?

As for Bosnian/Serbian/Croatian, this confusion set is more well-known (see [1]) and is notoriously quite difficult.

Let me know more details of your use case, and I'll try to offer some advice on how to increase accuracy.

[1] http://nlp.ffzg.hr/data/publications/nljubesi/tiedemann12-efficient.pdf

[plison]

Thanks for the answer! It seems the classification problem with Latvian comes from an encoding mistake on my part (used the wrong encoding format to decode the file). So everything looks fine now, I'm closing the issue!