Parallelizing pyBigWig.values() about pybigwig

[dpryan79]

Open the bigWig file inside extract_data().

[nicolazilio]

I tried. In principle that works in the sense that I don't get errors, but the problem is that, with that setup, as I increase the number of processes the computing time also increases.

[dpryan79]

For small regions that are near each other multithreading won't help you. Once you have 100kb or megabase regions the overhead of reading and decompressing is no longer rate limiting. In general, opening files inside worker forks is the only way to reliably access files in parallel with python.

[nicolazilio]

I have done some more research and it seems that, as you pointed out, increasing the number of processes indeed does not help a lot. However, the biggest reason for the slowdown that I was seeing was actually adding new rows to the pandas data frame 1000s of times. I changed that to writing to file directly and things improved A LOT.

Thanks again for the help again.

[dpryan79]

Glad you got things resolved!