struct.error and OverflowError about cort

[smartschat]

Unfortunately I do not know why there is a different behavior for these settings. :/

It's the size of all the features for an aggregation of documents. As far as I understood, the following happens: Python divides the data for multiprocessing into chunks, which then get processed. The results for the chunks are passed around using Python's pickle mechanism. If the results for the chunks are too big, the error occurs.

[smartschat]

This is a multiprocessing error which happens when too much data is passed around. I sometimes encountered it when experimenting with larger feature sets. A quick solution would be to disable multiprocessing. This, however, would vastly increase running time for feature extraction. Is this an option for you?

A more principled solution is a rewrite of the feature extraction code allowing for more efficient feature extraction/combination, but this will take some time.

[minhlab]

Hi @smartschat, thanks for answering. For some reason, the error doesn't occur when I run on a different machine: CentOS 7.2.1511, 62G RAM, 32 CPUs. Do you know why the difference?

BTW, is it the size of features of one document that exceeds 4GiB or is it an aggregation of multiple documents?