Cannot cut into N chunks about tomahawk

[mklarqvist]

Sorry for the confusion @slowkow . The partition has to be in the set N choose 2.
I will fix this limitation in the upcoming release

[slowkow]

Thank you!

I don't think this is a limitation, but it would be nice to add to the documentation so users can split work into jobs small enough for their computing environments.

Running on the Broad servers with the Sun Grid Engine job scheduler, I needed to split into 40186 jobs to get the jobs queued up quickly. Each job takes about 40 minutes or so, and most often less than 1G memory.

I brute forced my way to two numbers that work for me and split the work into a good number of jobs:

tomahawk calc -c 10012 <...>
tomahawk calc -c 40186 <...>

I guess I should be able to use R to find other numbers now (hooray!):

> choose(142, 2) + 1
10012

> choose(284, 2)
40186

I'm not sure why 10012 worked, because it should be 10011... maybe that was my mistake.

[mklarqvist]

That's cool. Just keep an eye on the amount of output data you generate. I've calculated genome-wide LD for 1000GP, HRC, and HGDP and the compressed and sorted outputs are ~800 GB, ~100 GB, and ~300 GB, respectively. These data will be available together with the upcoming publication.

I will look into why 142!2 + 1 worked. That is unexpected.