Training a new language on Windows doesn't work about langid.py

[saffsd]

Thanks for getting in touch! From the error message you get this is an OS-level issue - I've only tested the software on Linux. My guess is that on windows systems file descriptors can't be transmitted between processes, which breaks the technique I use to parallelize the learning.

Thanks for the reply. Can it be fixed on Windows?

[saffsd]

I have made some changes to the way that processes communicate in the parallelization, this method works for me under Win7. You can find the implementation in the windows-train branch. Please give it a try and let me know if it works for you. I will need to test it a bit to make sure it still works on Linux-based systems and produces the same output.

I tested the new code. I was able to generate a features file. I did however get an error at the end:

...
processed chunk (60/64) [723 terms]
processed chunk (61/64) [709 terms]
processed chunk (62/64) [710 terms]
processed chunk (63/64) [692 terms]
processed chunk (64/64) [716 terms]
selected 300 features
wrote features to "features"
Exception RuntimeError: RuntimeError('cannot join current thread',) in <Finalize object, dead> ignored

[saffsd]

The RuntimeError is a quirk with multiprocessing and the way it shuts down its worker pools. It shouldn't affect the result of the computation, but it's rather annoying so I've added calls to join() to ensure the pools are fully shut down before the program terminates.

Now it works without any errors. Thanks!

[saffsd]

I did a quick check and the updated training code appears to produce the same output as the old code on linux systems, so I have merged in back into master and I am closing this issue. Cheers!