Seeking advice regarding classification problem only present with Chinese about langid.py

[tripleee]

Are you sure the documents are in UTF-8? Windows software would often default to UTF-16 (if not some legacy code page).

[saffsd]

This definitely sounds like an encoding issue on the document side. When we trained langid.py we tried to include a representative sample of encodings, but I think the coverage for Chinese might be pretty poor. It's possible to retrain langid.py but this requires a bit of effort and training data. As @tripleee points out, windows often uses UTF-16, and quite a bit of the langid.py training data is in UTF-8. The easiest thing to try might be to transcode all documents (perhaps PDFMiner supports this directly? I'm not familiar) to UTF-8 and try again. Hope that helps!

[bittlingmayer]

For what it's worth, I see the opposite issue: bias towards Chinese

¡No! (only 24%)
‪#‎WCIT
Tʻagavorn apracʻ kenna
ՏԵՍԱՆՅՈՒԹ
#Cizre
#MustRead (only 77%)
Աֆրիկա (2nd, only 14%)

All are identified as Chinese, generally with > 98% probability.

Perhaps the Chinese data is actually all in the Latin alphabet? This should be the easiest language to keep separate, so it reeks of fundamental bug or preproc issue.

[bittlingmayer]

Pardon, looks like in most cases it is the result of invisible chars in dirty data. (But ՏԵՍԱՆՅՈՒԹ and ¡No! are clean, and ʻ in Tʻagavorn apracʻ kenna is not so exotic.