number of used threads by LTR_FINDER about edta

[oushujun]

Hi, some steps are not paralized. Let me know if there is any errors. For PiRATE, I have not tried yet. If you have the rice result, you can perform your own benchmark using the guidance at the bottom of the README.md file.

…

On Tue, Jan 14, 2020, 8:56 AM Kamil S. Jaron ***@***.***> wrote: Hello, I am running the whole pipeline on a huge server. I specified 64 cores, but for the last 4 hours, the program is using only 6 threads that running on ~20% each resulting in ~single core run. Wonder what might have gone wrong. I installed EDTA using conda (following instructions from README) and run it as follows perl EDTA/EDTA.pl -genome my_genome.fasta -species others -step all -anno 1 -threads 64 Thanks for making EDTA, it was a good twist in a benchmarking paper :-). By the way, did you try to compare EDTA with PiRATE? It also seems like a quite comprehensive pipeline, but I could find a comparison of the two. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#43?email_source=notifications&email_token=ABNX4NH7XGEF7DXTERMRETTQ5XVCTA5CNFSM4KGWKLCKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IGDWDJQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNX4NB6I23IH5RXBR3GFMLQ5XVCTANCNFSM4KGWKLCA> .

[KamilSJaron]

Hi, some steps are not paralized

I see, therefore the divide and conquer. In fact looking at the progress now, with TIRs it looks exactly the same. Perhaps it would be good to mention the single threaded bottlenecks in the readme, can I make a pull request?

Besides that, I do have ~1Mbp of LTRs annotated now :-).

PiRATE

If I will get to bench-marking of TE annotation pipeline, I will certainly take a look.

[oushujun]

Sorry, I thought you were talking about the entire EDTA pipeline. For LTR_FINDER_parallel, most of the heavy lifting jobs are parallized except the beginning and final stages, when the genome was split and combined, respectively. Some child process may take more time depending on the sequence. If you use the default settings it should be finished quickly. For PiRATE, if you have the result for the rice genome, I can benchmark it for you. Best, Shujun

…

On Wed, Jan 15, 2020, 8:38 AM Kamil S. Jaron ***@***.***> wrote: Hi, some steps are not paralized I see, therefore the divide and conquer. In fact looking at the progress now, with TIRs it looks exactly the same. Perhaps it would be good to mention the single threaded bottlenecks in the readme, can I make a pull request? Besides that, I do have ~1Mbp of LTRs annotated now :-). PiRATE If I will get to bench-marking of TE annotation pipeline, I will certainly take a look. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#43?email_source=notifications&email_token=ABNX4NCGECA47XZ2J22272LQ543ZNA5CNFSM4KGWKLCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJA6QFY#issuecomment-574744599>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNX4NALD2PUSEXQXZFS46DQ543ZNANCNFSM4KGWKLCA> .

[KamilSJaron]

I am running the whole EDTA pipeline, and once per while I peek at the CPU usage on the server and everytime time I take a look it's running a single core. As on the following screen shot

It seems that the script does spawn the threads, but they do not seem to run in parallel. I know that now the TIR annotation is using different programs, but the behavior seems to be the same for both - multiple threads spawned, but only single core used.

I mean this as a report of suspicious behaviour. If you think it's not a problem, feel free to ignore this issue.

I appreciate the effort of putting the pipeline together. Thank you for that.

[oushujun]

Hi Kamil, Python has its own way to create child processes and execute them. If its not reaching the ulimit it should be fine. Sometimes we see a process taking very low CPU and seems not running properly. It could be that it's running too fast and beyond the refreshing rate of the job monitor. So far I have not received bug reports for this behavior, but if you find it buggy please let me know. Best, Shujun

…

On Thu, Jan 16, 2020, 9:03 AM Kamil S. Jaron ***@***.***> wrote: I am running the whole EDTA pipeline, and once per while I peek at the CPU usage on the server and everytime time I take a look it's running a single core. As on the following screen shot [image: Screenshot from 2020-01-16 13-51-42] <https://user-images.githubusercontent.com/8181573/72530679-c704f800-3867-11ea-8eff-64afa9b1226e.png> It seems that the script does spawn the threads, but they do not seem to run in parallel. I know that now the TIR annotation is using different programs, but the behavior seems to be the same for both - multiple threads spawned, but only single core used. I mean this as a report of suspicious behaviour. If you think it's not a problem, feel free to ignore this issue. I appreciate the effort of putting the pipeline together. Thank you for that. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#43?email_source=notifications&email_token=ABNX4NASY2R5PIT576EFQQTQ6BZMVA5CNFSM4KGWKLCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJELZ7I#issuecomment-575192317>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNX4NC3WYQRLASHJNO763DQ6BZMVANCNFSM4KGWKLCA> .

[KamilSJaron]

Alright, then it's probably not a problem, I will close the issue for now/

I think rather simple test to verify the multicore utility would be to run exactly the same job with 1/2/4/8 cores a measure the speedup. I will explore and get back to you if I find anything.

[oushujun]

Yeah, I haven't tested on the CPU/memory efficiency yet. I would love to see your results. Using a small genome such as Arabidopsis should make the test faster. Best, Shujun

…

On Thu, Jan 16, 2020, 10:31 AM Kamil S. Jaron ***@***.***> wrote: Alright, then it's probably not a problem, I will close the issue for now/ I think rather simple test to verify the multicore utility would be to run exactly the same job with 1/2/4/8 cores a measure the speedup. I will explore and get back to you if I find anything. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#43?email_source=notifications&email_token=ABNX4NBYFLFQPEEADP6SGM3Q6CDVFA5CNFSM4KGWKLCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJEV7DA#issuecomment-575233932>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNX4NG7A5SNEX6XQGN3DXTQ6CDVFANCNFSM4KGWKLCA> .

[KamilSJaron]

Is it important to have a full genome? Would one drosophilla chromosome do the trick for efficiency benchmarking?
(that's how I intended to test it, just on a genome subsample)

[oushujun]

You need a big enough sample to collect enough instances of different TEs, otherwise the outcome will not be representing the whole genome. Subsample of >200 Mb should be enough, but that's bigger than the Ath genome already.