processing multiple inputs in parallel? about emg-viral-pipeline

[mberacochea]

Hi @shenwei356,

The pipeline was designed to process only input fasta at the time, but you should be able to run as many instances if VIRify as you want.

That error you are getting is related to Nextflow, not VIRIfy (as far as I can tell). Things you could try:

• Delete the cache and pull the pipeline again, rm -r /home/shenwei/.nextflow/assets/EBI-Metagenomics/emg-viral-pipeline and nextflow pull EBI-Metagenomics/emg-viral-pipeline (similar to nf-core/ampliseq#160)
• Not sure if related, but check that each instance of VIRify is using a different work directory

Cheers

[shenwei356]

Hi Martín, thanks for your rapid reply.

I've tried to clean and re-pull the pipeline, but the same error occurred. Besides, after trying with multiple instances, it would be failed to run with one instance. So I have to re-pull again to run with one instance.

Not sure if related, but check that each instance of VIRify is using a different work directory

Confirmed.

BTW the server has an old java:

NOTE: Nextflow is not tested with Java 1.8.0_332 -- It's recommended the use of version 11 up to 18

[hoelzer]

Hey @shenwei356 thanks for your interest in the pipeline! I'm not sure if I fully get what you are trying to do. But just in general and as also mentioned by @mberacochea:

• you can only start one nextflow pipeline from the same directory simultaneously. Nextflow is writing a hidden .nextflow.log file and therefore starting multiple nextflow commands from the same directory will cause errors.
• you can just give a multiFASTA as input

I'm not sure if the old JAVA might also cause issues.

[shenwei356]

you can only start one nextflow pipeline from the same directory simultaneously. Nextflow is writing a hidden .nextflow.log file and therefore starting multiple nextflow commands from the same directory will cause errors.

I see, but I specify the --workdir which point different places (where the inputs are). If it's true, the right way should be changing directory to where the input file is before running the pipline.

[hoelzer]

you can only start one nextflow pipeline from the same directory simultaneously. Nextflow is writing a hidden .nextflow.log file and therefore starting multiple nextflow commands from the same directory will cause errors.

I see, but I specify the --workdir which point different places (where the inputs are). If it's true, the right way should be changing directory to where the input file is before running the pipline.

Yeah --workdir or in short -w will not help you. This just changes where to working directories are written when running processes in the pipeline. Still there will be the hidden .nextflow.log file written to the directory from which you start the pipeline. No other pipeline can then be started from the same folder while the other one is still running and locks the .nextflow.log file - at least as far as I know. So yes, what you would need to do is: create a directory for each run, cd into that dir and then execute nextflow.

[shenwei356]

Thank you all!