tRNAscan-SE slow with multiple threads about bakta

[ohickl]

Hey, I just finished a Bakta prototype that splits the assembly into chunks and runs CMSCAN, etc., on each chunk. While I saw a slight speedup with more cores on my personal machine, on the cluster, even when loading the Bakta database into the node's memory, the processing speed would still decrease with a large number of cores (both tests were with unmodified Bakta).

With parallel chunk processing, the runtime for a sample with ~400K contigs and ~5Gbps length went from not finishing in a day to taking less than 2 hours for everything except the GenBank and EMBL output writing. The writing of these files takes extremely long for large files, so I added an option to skip it.

Now, I run this version with the database loaded into node memory and skipping EMBL/GenBank file writing, and it seems to run quite quickly even on large samples.

I'm not entirely sure how much potential network/latency issues on the cluster contribute to the original problem, but since the database is in the node memory, I assume it can't be too great of a factor.

If you have time, feel free to check my fork and let me know what you think in principle (it's pretty quick and dirty for now).

[oschwengers]

Thanks a lot @ohickl for checking and reporting!
I just ran a quick test on our test E.coli genome with 8 an 1 thread. But actually, running Bakta providing tRNAscan-SE only 1 core is ~10s slower than using 8.

It might be the case, that running Bakta using this large number of threads on a multiuser system invokes too much IO requests (network traffic between the machine and the maybe network-attached storage). Can you repeat these benchmarks exclusively using local storage?

[oschwengers]

Thank you very much! I'll take a look at it.