error message when run a test case about nextneopi

[riederd]

Hi, it seems that you do not have permissions to write to your resourcesBaseDir you specified. See:

Cannot write to ‘IEDB_MHC_I-3.1.4.tar.gz’ (Permission denied).

Can you show the output of the following commands:
id
ls -la your_resourcesBaseDir
(Please substitute your_resourcesBaseDir with the real path of that directory you configured)

[ryao-mdanderson]

Hello @riederd

You are right. my userid doesn't have write permission. my userid is ryao.
my resourceBaseDir configure in conf/param.conf is
resourcesBaseDir = "/rsrch3/scratch/reflib/REFLIB_data/nextneopi-1.4.0/"

ls -la /rsrch3/scratch/reflib/REFLIB_data/nextneopi-1.4.0/
total 42316629
drwxr-xr-x 6 root root 4096 Sep 28 17:18 .
drwxr-xr-x 70 root root 8192 Sep 29 09:35 ..
drwxr-xr-x 7 40005 50000 4096 Sep 29 09:16 databases
drwxr-xr-x 4 40005 50000 4096 May 3 2021 ExomeCaptureKits
-rw-r--r-- 1 root root 345348484 Jan 2 2023 IEDB_MHC_I-3.1.4.tar.gz
-rw-r--r-- 1 ryao rists 42662861027 Mar 22 2022 nextNEOpi_1.4_resources.tar.gz
-rw-r--r-- 1 ryao rists 323911161 Jul 27 09:24 nextNEOpi_testdata.tar.gz
drwxr-xr-x 7 40005 50000 4096 Mar 1 2022 references
drwxr-xr-x 2 40005 50000 4096 Jul 26 10:17 testdata

I am the system admin and have root privilege, I would like to store the resources in a shared location, so that our cluster users can also use this resource. So, if I change directory ownership to be root:root, allow group and others read permission, is okay?

The readme does not mention to install IEDB ahead of time until I ran the test and found out...
I manually download IEDB_MHC_I-3.1.4.tar.gz and untar it, configured iedb in location
cd mhc_i
bash -c ./configure

is this a right procedure?

[root@ldragon2 mhc_i]# ls -l /rsrch3/scratch/reflib/REFLIB_data/nextneopi-1.4.0/databases/iedb/mhc_i
total 389
-rwxrwxr-x 1 p_matlab p_matlab 19 Jan 2 2023 configure
-rw-rw-r-- 1 p_matlab p_matlab 11015 Jan 2 2023 Copenhagen_license.txt
drwxrwxr-x 3 p_matlab p_matlab 4096 Jan 2 2023 data
drwxrwxr-x 2 p_matlab p_matlab 4096 Jan 2 2023 examples
-rw-rw-r-- 1 p_matlab p_matlab 11839 Jan 2 2023 LIAI_license.txt
drwxrwxr-x 16 p_matlab p_matlab 4096 Jan 2 2023 method
-rw-rw-r-- 1 p_matlab p_matlab 134656 Jan 2 2023 mhc_list.xls
-rw-rw-r-- 1 p_matlab p_matlab 4795 Jan 2 2023 README
drwxrwxr-x 2 p_matlab p_matlab 4096 Sep 29 09:28 src

Now I wonder the ownership p_matlab.

Thank you for your help.

[ryao-mdanderson]

I forgot to mention, mhc_i/src/configure.py has an installation path limit 57 characters. In order to configure, I change the limit to 100, and it configured. Is this Okay? any reason for 57?

[ryao-mdanderson]

Hi @riederd

Update in my today's work:

1. I manually download IEDB_MHC_II-3.1.4.tar.gz, changed configure.py to allow a directory path length up to 100, then installed databases/mhc_i successfully

2. changed the ownership of /rsrch3/scratch/reflib/REFLIB_data/nextneopi-1.4.0/ to be a service account, with group and others read permission for the directory tree.

3. Because IEDB has been manually installed, created a non empty flag file databases/iedb/.iedb_install_ok.chck
   this is to avoid IEDB install triggered during the test run.

4, changed Tmp_dir in conf/param.config to make sure tmp directory has > 50GB

Then I ran the test case again, unfortunately, after 30 minutes run, still hit an issue to pull a singularity image

Error executing process > 'installVEPcache (installVEPcache)'
Caused by:
Failed to pull singularity image
command: singularity pull --name depot.galaxyproject.org-singularity-ensembl-vep-110.0--pl5321h2a3209d_0.img.pulling.1696016364914
https://depot.galaxyproject.org/singularity/ensembl-vep:110.0--pl5321h2a3209d_0 > /dev/null
status : 143

At this point, the test run is not successful.

Thank you for you help.

[riederd]

Hi,

• IEDB installation:
  it is not intended that users install IEDB manually. This is all handled internally by nextNEOpi.
  You could give a temporary write access to your non root account for the resources dir and then run the pipeline once so that all data will be correctly installed. Afterwards you may change the permissions for the resources dir to read only.

• VEP singularity image:
  This might be a temporary error due to some timeout or whatsoever. It looks like that singularity is unable to correctly pull the VEP image from the galaxyproject server.

Can you post the contents (.tar.gz) of the work dir of the failed process?

Did you retry? What happens when you run the process manually? e.g.:

cd <workdir of the failed process>
bash .command.run

[riederd]

Any feedback on this?

[ryao-mdanderson]

Hi @riederd Dietmar,

I am very sorry for a late reply. One of the reasons is I continued to hit singularity pull ensembl-vep. Not only during the nextflow run, but also with manually command pull. With this error could not be resolved, I am stuck on nextflow test run.

[ryao@myserver ~]$ singularity pull depot.galaxyproject.org-singularity-ensembl-vep-110.0--pl5321h2a3209d_0.img.pulling.1696016364914 https://depot.galaxyproject.org/singularity/ensembl-vep:110.0--pl5321h2a3209d_0

INFO: Downloading network image
629.7MiB / 991.6MiB [=================================================>----------------------------] 64 % 358.3 KiB/s 17m14s
FATAL: net/http: request canceled (Client.Timeout exceeded while reading body)

The above is an example of my singularity pull. By the way, are you able to pull on your server?

For IEDB installation. I believe your application has handled this automatically, since I was able to manually installed IEDB, I was curious to get nextflow test work first. Therefore I did not retry, partially due to the reference directory in a shared directory that controlled by GPFS file system with access control. Simply change the write access for a non root user will not work.

Thank you for your help.
Rong Yao

[riederd]

Hi, it seems that you are having some sort of (temporary?) network issues. I just tried to pull the image and it went smoothly, i took 2 minutes.

You might try to set a longer timeout for singularity image pulls, e.g. at line:

nextNEOpi/conf/profiles.config

Line 91 in fe7b21c

add the following:

singularity.pullTimeout = 3600

[ryao-mdanderson]

Hi @riederd,

I followed your suggestion to add singularity.pullTimeout = 3600 in conf/profile.config and rerun the nextNEOpi test data, this time it exited quickly due to pull fastqc image. I have attached a file with the full output in this message.

I also get a try to manually singularity pull ensembl-vep-110.0 again, after 17 minutes, it exited with same error as I reported.

I wonder you get to pull the image in 2 minutes, is due to physical location? I guess depot.galaxyproject.org hosted in Europe and I worked in Texas USA.

I feel it is frustrating to get it work.
nextNEOpi-test.txt

Thank you ,
Rong Yao

[riederd]

I'm sorry that you are running into these issues, which apparently are caused by something not specific to nextNEOpi, since even pulling the image manually fails. Unfortunately I can not reproduce them, so it gets difficult for me to pinpoint the root cause.

Are there any limits set on the system you are using, e.g. memory, cputime

[ryao-mdanderson]

@riederd you have been very helpful already, thank you.

Today, I finally successfully manually pull ensembl-vep image on a HPC cluster node, it took ~30 minutes; Comparing to your pull in 2 minutes, a significant difference.

I am waiting for a firewall open for https://apps-01.i-med.ac.at-images on the cluster node. I will test the nextflow again when this is ready.

Cheers,
Rong

[riederd]

I'm closing this now