Error executing process > 'gatherMutect2VCFs (sample2)' about nextneopi

[zktuong]

Hi, the --containall run option was removed from the default in nextNEOpi see ebfb100

If you need it for NeoFuse to work, you may put it into:

nextNEOpi/conf/process.config

Lines 169 to 172 in 71046cf

	withName:Neofuse {
	container = 'https://apps-01.i-med.ac.at/images/singularity/NeoFuse_dev_92712e07.sif'
	cpus = 10
	}

e.g.:

     withName:Neofuse {
        container = 'https://apps-01.i-med.ac.at/images/singularity/NeoFuse_dev_92712e07.sif'
        cpus = 10
        containerOptions = "--containall"
    }

Let me know if this works for you.

thanks! yes this works for me now. Both gatherMutect2VCFs and Neofuse are working now in a single run with the following modifications:

since params.config has includeConfig './process.config' above, so i modified the last parts of params.config to:

singularity {
    enabled = true
    autoMounts = true
    runOptions =  process.containerOptions + " -H " + params.singularityTmpMount + " -B " +  params.singularityAssetsMount + " -B " + params.singularityTmpMount + " -B " + params.resourcesBaseDir + params.singularityHLAHDmount + " -B " + params.databases.IEDB_dir + ":/opt/iedb" + " -B " + params.databases.MHCFLURRY_dir + ":/opt/mhcflurry_data"
}

and process.config to add:

    // default singularity run options
    containerOptions = "--no-home"

    withName:Neofuse {
        container = 'https://apps-01.i-med.ac.at/images/singularity/NeoFuse_dev_92712e07.sif'
        cpus = 60
        containerOptions = "--no-home --containall"
    }

Just need to see if it runs through the rest ok.

[riederd]

This is a GATK error. Just to be sure, can you try to manually reproduce it by:

cd /QRISdata/Q6373/results/results_1/work/32/4f5f39ab6af08ba9cc90eed3361958
bash .command.run

[amoschoomy]

Hi sorry for the late reply, I've actually rerun it but I ran into the same error but on a different work directory. And I ran what you said and run into ln: failed to create symbolic link 'sample1_0039-scattered.interval_list.vcf.gz': File exists

work.tar.gz

I've uploaded the work directory for your use

[riederd]

It seems that /QRISdata/Q6373/results/results_1/work/f3/1a75384eb71df85c5cac8ba99d4b6c/sample1_0015-scattered.interval_list-f1r2.tar.gz does not exist.
Can you change into /QRISdata/Q6373/results/results_1/work/f3/1a75384eb71df85c5cac8ba99d4b6c/ and post the contents of the directory as tar.gz.?

[amoschoomy]

Hi, I've checked it and the file existed but nevertheless here are the directory contents
work2.tar.gz

Thanks!

[riederd]

this is strange. May I ask you to try:

cd /QRISdata/Q6373/results/results_1/work/ae/e43079d0e13743a5bd94b0f958e0e7
singularity exec -B /QRISdata/Q6373/results/results_1/work -B /home/uqachoo1/nextNEOpi-1.4.0/bin --no-home --containall -H /scratch/temp/5890914 -B /home/uqachoo1/nextNEOpi-1.4.0/assets -B /scratch/temp/5890914 -B /QRISdata/Q6196/resources -B /home/uqachoo1/hlahd.1.7.0 -B /QRISdata/Q6196/resources/databases/iedb:/opt/iedb -B /QRISdata/Q6196/resources/databases/mhcflurry_data:/opt/mhcflurry_data /QRISdata/Q6373/results/results_1/work/singularity/apps-01.i-med.ac.at-images-singularity-nextNEOpi_1.4.0_f894e892.sif /bin/bash

and then

gatk LearnReadOrientationModel \
    --tmp-dir /scratch/temp/5890914 \
    -I sample1_0039-scattered.interval_list-f1r2.tar.gz -I sample1_0003-scattered.interval_list-f1r2.tar.gz -I sample1_0017-scattered.interval_list-f1r2.tar.gz -I sample1_0029-scattered.interval_list-f1r2.tar.gz -I sample1_0025-scattered.interval_list-f1r2.tar.gz -I sample1_0009-scattered.interval_list-f1r2.tar.gz -I sample1_0002-scattered.interval_list-f1r2.tar.gz -I sample1_0027-scattered.interval_list-f1r2.tar.gz -I sample1_0036-scattered.interval_list-f1r2.tar.gz -I sample1_0021-scattered.interval_list-f1r2.tar.gz -I sample1_0028-scattered.interval_list-f1r2.tar.gz -I sample1_0012-scattered.interval_list-f1r2.tar.gz -I sample1_0004-scattered.interval_list-f1r2.tar.gz -I sample1_0033-scattered.interval_list-f1r2.tar.gz -I sample1_0014-scattered.interval_list-f1r2.tar.gz -I sample1_0019-scattered.interval_list-f1r2.tar.gz -I sample1_0011-scattered.interval_list-f1r2.tar.gz -I sample1_0038-scattered.interval_list-f1r2.tar.gz -I sample1_0015-scattered.interval_list-f1r2.tar.gz -I sample1_0005-scattered.interval_list-f1r2.tar.gz -I sample1_0018-scattered.interval_list-f1r2.tar.gz -I sample1_0026-scattered.interval_list-f1r2.tar.gz -I sample1_0035-scattered.interval_list-f1r2.tar.gz -I sample1_0030-scattered.interval_list-f1r2.tar.gz -I sample1_0016-scattered.interval_list-f1r2.tar.gz -I sample1_0001-scattered.interval_list-f1r2.tar.gz -I sample1_0013-scattered.interval_list-f1r2.tar.gz -I sample1_0008-scattered.interval_list-f1r2.tar.gz -I sample1_0022-scattered.interval_list-f1r2.tar.gz -I sample1_0024-scattered.interval_list-f1r2.tar.gz -I sample1_0034-scattered.interval_list-f1r2.tar.gz -I sample1_0006-scattered.interval_list-f1r2.tar.gz -I sample1_0020-scattered.interval_list-f1r2.tar.gz -I sample1_0010-scattered.interval_list-f1r2.tar.gz -I sample1_0031-scattered.interval_list-f1r2.tar.gz -I sample1_0037-scattered.interval_list-f1r2.tar.gz -I sample1_0000-scattered.interval_list-f1r2.tar.gz -I sample1_0007-scattered.interval_list-f1r2.tar.gz -I sample1_0032-scattered.interval_list-f1r2.tar.gz -I sample1_0023-scattered.interval_list-f1r2.tar.gz \
    -O sample1_read-orientation-model.tar.gz

and

gatk LearnReadOrientationModel \
    --tmp-dir /scratch/temp/5890914 \
    -I  sample1_0014-scattered.interval_list-f1r2.tar.gz \
    -O int_0014_read-orientation-model.tar.gz

[zktuong]

hi, i've been experiencing this as well. I ran it manually as you suggested and I chased it down to --containall was stuffing it up in the singularity exec step. removing it manually made it work.

i'm not well versed with nextflow and nextNEOpi so could you tell me where should i disable this??

[zktuong]

ok sorry now i'm getting a better sense of what's happening.

for clarity, instead of the original code:

nextNEOpi/conf/params.config

Lines 277 to 281 in fe7b21c

	singularity {
	enabled = true
	autoMounts = true
	runOptions = "--no-home" + " -H " + params.singularityTmpMount + " -B " + params.singularityAssetsMount + " -B " + params.singularityTmpMount + " -B " + params.resourcesBaseDir + params.singularityHLAHDmount + " -B " + params.databases.IEDB_dir + ":/opt/iedb" + " -B " + params.databases.MHCFLURRY_dir + ":/opt/mhcflurry_data"
	}

The copy of my conf/params.config was the following as i had previous seen this #25

singularity {
    enabled = true
    autoMounts = true
    runOptions =  "--no-home --containall" + " -H " + params.singularityTmpMount + " -B " +  params.singularityAssetsMount + " -B " + params.singularityTmpMount + " -B " + params.resourcesBaseDir + params.singularityHLAHDmount + " -B " + params.databases.IEDB_dir + ":/opt/iedb" + " -B " + params.databases.MHCFLURRY_dir + ":/opt/mhcflurry_data"
}

so it seems that having --containall could make the NeoFuse works but not the gatherMutect2VCFs step.

Is there a more permanent solution? seems a bit tedious to have to run the pipeline twice (without and with --containall) as the workaround?

[riederd]

Hi,
the --containall run option was removed from the default in nextNEOpi see ebfb100

If you need it for NeoFuse to work, you may put it into:

nextNEOpi/conf/process.config

Lines 169 to 172 in 71046cf

	withName:Neofuse {
	container = 'https://apps-01.i-med.ac.at/images/singularity/NeoFuse_dev_92712e07.sif'
	cpus = 10
	}

e.g.:

     withName:Neofuse {
        container = 'https://apps-01.i-med.ac.at/images/singularity/NeoFuse_dev_92712e07.sif'
        cpus = 10
        containerOptions = "--containall"
    }

Let me know if this works for you.

[amoschoomy]

Hi @riederd I've tried executing it manually and it worked (I had to specify a different $TMPDIR on that manual command as i do not have access to the temp directory at runtime).

I then reexecuted the pipeline with the full stack trace of GATK enabled and now I'm getting the same issue. Apparently I have no space left on my device? Which is weird because I've allocated 5TB on my shared project space and $TMPDIR is setup and allocated automatically by the HPC system at the submission of jobs (which would be sufficient for most of the heavy jobs?) I'm not sure how much more do GATK needs in regards to this or is there a way to reduce the space usage?

Command error:
  17:39:44.852 INFO  IOUtils - Extracting file: ./sample1_tumor.alt_table
  17:39:44.857 INFO  IOUtils - Extracting data from archive: file:///scratch/project_mnt/S0142/results/results_1/work/84/389049f48f5f9b757eb2fcc716e0fa/sample1_0028-scattered.interval_list-f1r2.tar.gz
  17:39:44.858 INFO  IOUtils - Extracting file: ./sample1_tumor.alt_histogram
  17:39:44.859 INFO  IOUtils - Extracting file: ./sample1_tumor.ref_histogram
  17:39:44.859 INFO  IOUtils - Extracting file: ./sample1_tumor.alt_table
  17:39:44.865 INFO  IOUtils - Extracting data from archive: file:///scratch/project_mnt/S0142/results/results_1/work/84/389049f48f5f9b757eb2fcc716e0fa/sample1_0032-scattered.interval_list-f1r2.tar.gz
  17:39:44.867 INFO  IOUtils - Extracting file: ./sample1_tumor.alt_histogram
  17:39:44.867 INFO  IOUtils - Extracting file: ./sample1_tumor.ref_histogram
  17:39:44.868 INFO  IOUtils - Extracting file: ./sample1_tumor.alt_table
  17:39:44.869 INFO  LearnReadOrientationModel - Shutting down engine
  [October 16, 2023 at 5:39:44 PM UTC] org.broadinstitute.hellbender.tools.walkers.readorientation.LearnReadOrientationModel done. Elapsed time: 0.00 minutes.
  Runtime.totalMemory()=335544320
  ***********************************************************************
  
  A USER ERROR has occurred: Could not extract data from: file:///scratch/project_mnt/S0142/results/results_1/work/84/389049f48f5f9b757eb2fcc716e0fa/sample1_0032-scattered.interval_list-f1r2.tar.gz
  
  ***********************************************************************
  org.broadinstitute.hellbender.exceptions.UserException: Could not extract data from: file:///scratch/project_mnt/S0142/results/results_1/work/84/389049f48f5f9b757eb2fcc716e0fa/sample1_0032-scattered.interval_list-f1r2.tar.gz
  	at org.broadinstitute.hellbender.utils.io.IOUtils.extractTarGz(IOUtils.java:458)
  	at org.broadinstitute.hellbender.utils.io.IOUtils.extractTarGz(IOUtils.java:434)
  	at org.broadinstitute.hellbender.tools.walkers.readorientation.LearnReadOrientationModel.lambda$doWork$1(LearnReadOrientationModel.java:86)
  	at java.base/java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:104)
  	at java.base/java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:617)
  	at org.broadinstitute.hellbender.tools.walkers.readorientation.LearnReadOrientationModel.doWork(LearnReadOrientationModel.java:86)
  	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:149)
  	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:198)
  	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:217)
  	at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
  	at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
  	at org.broadinstitute.hellbender.Main.main(Main.java:289)
  Caused by: java.io.IOException: No space left on device
  	at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
  	at java.base/sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:62)
  	at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132)
  	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:97)
  	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:67)
  	at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:288)
  	at java.base/java.nio.channels.Channels.writeFullyImpl(Channels.java:74)
  	at java.base/java.nio.channels.Channels.writeFully(Channels.java:96)
  	at java.base/java.nio.channels.Channels$1.write(Channels.java:171)
  	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1310)
  	at org.apache.commons.io.IOUtils.copy(IOUtils.java:978)
  	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1282)
  	at org.apache.commons.io.IOUtils.copy(IOUtils.java:953)
  	at org.broadinstitute.hellbender.utils.io.IOUtils.extractFilesFromArchiveStream(IOUtils.java:523)
  	at org.broadinstitute.hellbender.utils.io.IOUtils.extractTarGz(IOUtils.java:455)
  	... 11 more
  Using GATK jar /opt/gatk/gatk-package-4.4.0.0-local.jar
  Running:
      java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx50G -XX:ParallelGCThreads=60 -DGATK_STACKTRACE_ON_USER_EXCEPTION=true -jar /opt/gatk/gatk-package-4.4.0.0-local.jar LearnReadOrientationModel --tmp-dir /scratch/temp/5976135 -I sample1_0034-scattered.interval_list-f1r2.tar.gz -I sample1_0025-scattered.interval_list-f1r2.tar.gz -I sample1_0036-scattered.interval_list-f1r2.tar.gz -I sample1_0001-scattered.interval_list-f1r2.tar.gz -I sample1_0038-scattered.interval_list-f1r2.tar.gz -I sample1_0020-scattered.interval_list-f1r2.tar.gz -I sample1_0023-scattered.interval_list-f1r2.tar.gz -I sample1_0016-scattered.interval_list-f1r2.tar.gz -I sample1_0010-scattered.interval_list-f1r2.tar.gz -I sample1_0022-scattered.interval_list-f1r2.tar.gz -I sample1_0019-scattered.interval_list-f1r2.tar.gz -I sample1_0004-scattered.interval_list-f1r2.tar.gz -I sample1_0028-scattered.interval_list-f1r2.tar.gz -I sample1_0032-scattered.interval_list-f1r2.tar.gz -I sample1_0018-scattered.interval_list-f1r2.tar.gz -I sample1_0011-scattered.interval_list-f1r2.tar.gz -I sample1_0029-scattered.interval_list-f1r2.tar.gz -I sample1_0008-scattered.interval_list-f1r2.tar.gz -I sample1_0015-scattered.interval_list-f1r2.tar.gz -I sample1_0031-scattered.interval_list-f1r2.tar.gz -I sample1_0024-scattered.interval_list-f1r2.tar.gz -I sample1_0035-scattered.interval_list-f1r2.tar.gz -I sample1_0003-scattered.interval_list-f1r2.tar.gz -I sample1_0030-scattered.interval_list-f1r2.tar.gz -I sample1_0000-scattered.interval_list-f1r2.tar.gz -I sample1_0039-scattered.interval_list-f1r2.tar.gz -I sample1_0006-scattered.interval_list-f1r2.tar.gz -I sample1_0033-scattered.interval_list-f1r2.tar.gz -I sample1_0021-scattered.interval_list-f1r2.tar.gz -I sample1_0012-scattered.interval_list-f1r2.tar.gz -I sample1_0007-scattered.interval_list-f1r2.tar.gz -I sample1_0014-scattered.interval_list-f1r2.tar.gz -I sample1_0037-scattered.interval_list-f1r2.tar.gz -I sample1_0026-scattered.interval_list-f1r2.tar.gz -I sample1_0009-scattered.interval_list-f1r2.tar.gz -I sample1_0005-scattered.interval_list-f1r2.tar.gz -I sample1_0013-scattered.interval_list-f1r2.tar.gz -I sample1_0027-scattered.interval_list-f1r2.tar.gz -I sample1_0017-scattered.interval_list-f1r2.tar.gz -I sample1_0002-scattered.interval_list-f1r2.tar.gz -O sample1_read-orientation-model.tar.gz

[riederd]

I'll implement these changes, and close the issue.
Feel free to reopen if it needs more discussion