Parallel causes a net slowdown about parallel

[grosser]

they don't do anything special ... try doing .each or calling .size on the result to see if it's the process-to-process overhead

[grosser]

could try ractors ...

[michaeleisel]

With .each, on a subset of those JSON files (16 files) and after running repeatedly to warm the caches, I get 2.8 seconds with normal serial execution and 3.6 seconds with parallel. So, the difference is not as drastic but still there. With ractors, I see similar decreased performance.

[grosser]

the only thing I could imagine doing that is disk io ...
maybe read the files into memory first and then start the blocks ?
(also make sure to return nil or the count from the blocks just in case)

[reitzig]

For spinning disks, random access is relatively slower compared to linear scan. You are converting N linear scans into random access patterns across the same data volume by parallelizing.
Not sure if it's the same for SSDs, but any form of caching the hardware or OS do will be similarly affected.

[michaeleisel]

I wonder if we could improve this by improving serialization speed when consolidating the objects produced by child processes by .map?