optimize `sort` for large data sets about redundans

[lpryszcz]

Hi Thomas, Honestly, I've never tried it on such big dataset, so I'm super glad for the info!
You can try sorting in memory ie. sort -S 80% or sort -S 200G depending on amount of memory you have. For that change fasta2homozygous.py:

• line 28:

    cmd += "awk '$10!=$14 && $11>=$15' | sort -k11nr,11 -k12n,12 -k13nr,13 | gzip > %s.psl.gz"%fasta

to

    cmd += "awk '$10!=$14 && $11>=$15' | sort -S %s -k11nr,11 -k12n,12 -k13nr,13 | gzip > %s.psl.gz"%("80%", fasta)

• and line 62:

    cmd2 = "awk '$10!=$14 && $11>=$15' %s*.psl | sort -k11nr,11 -k12n,12 -k13nr,13 | gzip > %s.psl.gz"%(fasta, fasta)

to

    cmd2 = "awk '$10!=$14 && $11>=$15' %s*.psl | sort -S %s -k11nr,11 -k12n,12 -k13nr,13 | gzip > %s.psl.gz"%(fasta, "80%", fasta)

For more details have a look here: http://stackoverflow.com/a/16927657/632242
I'll think how to improve it in the production.

[lpryszcz]

btw: I've pushed these changes, so you can also try updating your code git pull origin master

[thackl]

Jep, I was thinking along those lines as well. I will try to sort the psl files manually first and try to find some optimized settings. I'll let you know, how it goes.

And one more thing - are the psl files the result of the blat or last alignments. Once I get things sorted, I'd like to try to use the sorted psl file directly and skip the actual alignment step - it took quite some time to get them

[thackl]

I've already changed those lines anyway - needed to include -T because my /tmp drive was to small...

[lpryszcz]

It depends on your settings:

• BLAT for --identity >=0.85
• LAST --identity <0.85

Default --identity is 0.51, thus LAST is used. Last is multi-threaded, BLAT is not. In my tests LAST was much more sensitive, but also ~4x slower. Providing multi-threaded BLAT function is on my list, but didn't have time to do it yet...

https://github.com/lpryszcz/redundans#why-does-redundans-use-two-similarity-search-algorithms-blat--last

[thackl]

Ah, makes sense. identity=0.51, so last.. Might make sense to try higher idy and blat first, especially on this large data set

[thackl]

So, yes, sorting in memory definitely solves the problem, now awk and later gzip runs at 100%. Would it be possible to setup a pass-through option for the sort parameters so one could run something like redundans.py --sortopt "-T /some/where -S 30%"?

[lpryszcz]

it is solved in the newest version (-S / --sortopt added)