Running EDTA efficiently on a cluster about edta

[oushujun]

Hi Dan,

Yes the devide and conquer step was designed for this purpose, you can run the three jobs at the same time in the same work dir. However, you may want to differ the start time for a minute or two because the current initial step will convert the genome IDs. Doing the conversion at the same time may cause problems. Each of these steps should be finished within a week, and the remaining steps should be done within a couple days.

Please let me know if you encounter any problems.

Shujun

[DanJeffries]

Thanks Shujun,

I indeed encountered the issue with the step to convert the genome IDs, starting 1 job and waiting for the other two until the modified genome was created worked. Thanks!

I am now running this, although I am running into problems with lack of memory for the LTR step (suffixerator gets killed) and the Helitron step (run_helitron_scanner.sh gets killed). I initially allocated 16 cpus and 64Gb for these jobs, so trying with 128Gb now and will update final run times and resources when I have them.

For the TIR run I got some errors unrelated to memory though . . . perhaps you have an idea if this is important?

/scratch/axiom/FAC/FBM/DEE/nperrin/rana_genome/EDTA/EDTA/bin/TIR-Learner2.4/TIR-Learner2.4.sh: line 245: rsync: command not found
/scratch/axiom/FAC/FBM/DEE/nperrin/rana_genome/EDTA/EDTA/bin/TIR-Learner2.4/TIR-Learner2.4.sh: line 249: rsync: command not found
/scratch/axiom/FAC/FBM/DEE/nperrin/rana_genome/EDTA/EDTAenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is depre
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/scratch/axiom/FAC/FBM/DEE/nperrin/rana_genome/EDTA/EDTAenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is depre
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])

Thanks

Dan

[oushujun]

Good to know it's working! I may modify the script to make different runs adaptatable to each other. For the rsync error, I assume all systems have rsync and apparently it's not. It won't affect the results because it's a backup step. Let me think about alternatives. Thanks for the report. Shujun

…

On Sun, Feb 2, 2020, 1:22 AM Dan Jeffries ***@***.***> wrote: Thanks Shujun, I indeed encountered the issue with the step to convert the genome IDs, starting 1 job and waiting for the other two until the modified genome was created worked. Thanks! I am now running this, although I am running into problems with lack of memory for the LTR step (suffixerator gets killed) and the Helitron step (run_helitron_scanner.sh gets killed. I initially allocated 16 cpus and 64Gb for these jobs, so trying with 128Gb now and will update final run times and resources when I have them. For the TIR run I got some errors unrelated to memory though . . . perhaps you have an idea if this is important? """ /scratch/axiom/FAC/FBM/DEE/nperrin/rana_genome/EDTA/EDTA/bin/TIR-Learner2.4/ TIR-Learner2.4.sh: line 245: rsync: command not found /scratch/axiom/FAC/FBM/DEE/nperrin/rana_genome/EDTA/EDTA/bin/TIR-Learner2.4/ TIR-Learner2.4.sh: line 249: rsync: command not found /scratch/axiom/FAC/FBM/DEE/nperrin/rana_genome/EDTA/EDTAenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is depre _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /scratch/axiom/FAC/FBM/DEE/nperrin/rana_genome/EDTA/EDTAenv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is depre _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) """ Thanks Dan — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#52?email_source=notifications&email_token=ABNX4NGI63AHD2CFZN5HJL3RAZYBVA5CNFSM4KODI5LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKRPX7A#issuecomment-581106684>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNX4NGWLJXI72KTHBO5RBDRAZYBVANCNFSM4KODI5LA> .

[oushujun]

@DanJeffries rsync has been replaced by cp with a for loop as reflected in the v1.7.9 update. You don't need this update unless TIR results are not produced.