Error at the 4-quiver step of pb-assembly about pbbioconda

[pb-cdunn]

+ samtools faidx /data2/avelt/Assembly_amurensis/4-quiver/quiver-split/./refs/000153Fp01_00001/ref.fa
[faidx] Could not build fai index /data2/avelt/Assembly_amurensis/4-quiver/quiver-split/./refs/000153Fp01_00001/ref.fa.fai

Could you add that ref.fa to dropbox? Or is that file empty? If it's short, just post it here.

[pb-cdunn]

After you look at that ref.fa, note that you are using an old pb-assembly/pb-falcon. Try updating, re-post conda list, and re-run.

But look at ref.fa first. We are curious about the actual crash.

[a-velt]

Hi,

Thank you for your help. There is just the header in the ref.fa file :
more /data2/avelt/Assembly_amurensis/4-quiver/quiver-split/./refs/000153Fp01_00001/ref.fa

>000153Fp01_00001

I launched:
conda update pb-assembly

And now the conda list :

# packages in environment at /cm/shared/apps/FALCON_12_09_2018:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py27_0
avro                      1.8.0                    py27_0    bioconda
bcftools                  1.9                  h4da6232_0    bioconda
bedtools                  2.27.1               he941832_2    bioconda
blas                      1.0                         mkl
blasr                     5.3.2                hac9d22c_3    bioconda
blasr_libcpp              5.3.1                hac9d22c_2    bioconda
bwa                       0.7.17               ha92aebf_3    bioconda
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.8.24            ha4d7672_0    conda-forge
certifi                   2018.8.24             py27_1001    conda-forge
cffi                      1.11.5           py27he75722e_1
chardet                   3.0.4                    py27_1
conda                     4.5.11                   py27_0    conda-forge
conda-env                 2.6.0                         1
cryptography              2.3.1            py27hc365091_0
curl                      7.61.0               h93b3f91_2    conda-forge
cython                    0.28.5           py27hfc679d8_0    conda-forge
decorator                 4.3.0                      py_0    conda-forge
enum34                    1.1.6                    py27_1
future                    0.16.0                   py27_2    conda-forge
futures                   3.2.0                    py27_0
genomicconsensus          2.3.2                    py27_1    bioconda
h5py                      2.8.0            py27hb794570_1    conda-forge
hdf5                      1.10.2               hc401514_2    conda-forge
htslib                    1.7                           0    bioconda
idna                      2.7                      py27_0
intel-openmp              2018.0.3                      0
ipaddress                 1.0.22                   py27_0
iso8601                   0.1.12                     py_1    conda-forge
krb5                      1.14.6                        0    conda-forge
libdeflate                1.0                  h470a237_0    bioconda
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 8.2.0                hdf63c60_1
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libssh2                   1.8.0                h5b517e9_2    conda-forge
libstdcxx-ng              8.2.0                hdf63c60_1
linecache2                1.0.0                    py27_0    conda-forge
minimap2                  2.12                 ha92aebf_0    bioconda
mkl                       2018.0.3                      1
mkl_fft                   1.0.6                    py27_0    conda-forge
mkl_random                1.0.1                    py27_0    conda-forge
mummer                    3.23                    pl526_6    bioconda
mummer4                   4.0.0beta2      pl526hfc679d8_3    bioconda
ncurses                   6.1                  hf484d3e_0
networkx                  2.1                        py_1    conda-forge
nim-falcon                0.0.0                         0    bioconda
numpy                     1.15.1           py27h1d66e8a_0
numpy-base                1.15.1           py27h81de0dd_0
openssl                   1.0.2p               h470a237_0    conda-forge
pb-assembly               0.0.1                    py27_0    bioconda
pb-dazzler                0.0.0                h470a237_0    bioconda
pb-falcon                 0.2.3            py27ha92aebf_0    bioconda
pbalign                   0.3.1                    py27_0    bioconda
pbbam                     0.18.0               h1310cd9_1    bioconda
pbcommand                 1.1.1            py27h24bf2e0_1    bioconda
pbcore                    1.5.1                    py27_1    bioconda
perl                      5.26.2               h470a237_0    conda-forge
pip                       10.0.1                   py27_0
pycosat                   0.6.3            py27h14c3975_0
pycparser                 2.18                     py27_1
pyopenssl                 18.0.0                   py27_0
pysam                     0.14.1           py27hae42fb6_1    bioconda
pysocks                   1.6.8                    py27_0
python                    2.7.15               h1571d57_0
python-consensuscore      1.1.1            py27h02d93b8_1    bioconda
python-consensuscore2     3.1.0                    py27_1    bioconda
python-edlib              1.2.3            py27h470a237_1    bioconda
python-intervaltree       2.1.0                      py_0    bioconda
python-msgpack            0.5.6            py27h470a237_0    bioconda
python-sortedcontainers   2.0.4                      py_0    bioconda
pytz                      2018.5                     py_0    conda-forge
readline                  7.0                  h7b6447c_5
requests                  2.19.1                   py27_0
ruamel_yaml               0.15.46          py27h14c3975_0
samtools                  1.9                  h8ee4bcc_1    bioconda
setuptools                40.2.0                   py27_0
six                       1.11.0                   py27_1
sqlite                    3.24.0               h84994c4_0
tk                        8.6.8                hbc83047_0
traceback2                1.4.0                    py27_0    conda-forge
unittest2                 1.1.0                      py_0    conda-forge
urllib3                   1.23                     py27_0
wheel                     0.31.1                   py27_0
xz                        5.2.4                h470a237_1    conda-forge
yaml                      0.1.7                had09818_2
zlib                      1.2.11               ha838bed_2

I will launch the 4-quiver step with the new version of pb-assembly/pb-falcon. I will keep you in touch.

Best,
Amandine

[isovic]

Hi Amandine,
The empty sequence in the ref.fa is peculiar...
Could you please zip and upload the contents of the 3-unzip/2-htigs/chunk_000153F folder? Hopefully it shouldn't be too big, and we would like to take a look.

Best regards,
Ivan.

[a-velt]

Here the zip file :
chunk_000153F.zip

Thank you for your help.
Best,
Amandine

[a-velt]

I get the same error with the last version of pb-assembly. Maybe I need to restart the 3-unzip step with the new version too? Because here I had launched only stage 4. I will test.

[isovic]

Thank you for sharing the data!
It seems that for some reason a zero-length haplotig is being extracted, which in turn causes samtools to exit non-zero. We're looking into it, but for now, you could try manually removing it from 3-unzip/all_h_ctg.fa and re-launching the 4-quiver stage.

It could also be useful for our debugging if you could also zip and send the 3-unzip/0-phasing/000153F folder?
Best regards,
Ivan

[a-velt]

Here the 3-unzip/0-phasing/000153F folder :

000153F.zip

I will remove it from 3-unzip/all_h_ctg.fa file and try to run 4-quiver stage, I keep you in touch.

Best,
Amandine

[a-velt]

It works by removing the sequence from 3-unzip/all_h_ctg.fa !

[isovic]

To anyone else running into this same issue:
In the latest version of Unzip we fixed a bug that could have caused empty haplotigs to be written to the FASTA file, but also provide a safeguard to prevent such haplotigs and primary contigs to be written.
This is now available in falcon-unzip 1.1.4 available in pb-falcon>=0.2.4 and pb-assembly>=0.0.2.
Note: this fix applies only if the 3-unzip stage is re-run. If the rerun process is costly and you have completed it using an older version, please use the same workaround as @a-velt and manually filter the empty sequences from 3-unzip/all_h_ctg.fa.

[NicMAlexandre]

I have run into this issue as well, updated pb-assembly, falcon unzip, and pb-falcon and rerun the process from the 3-unzip step. As above, my errors still occur in the quiver-run and quiver-chunks step.

I looked for a reference file in the 4-quiver folder, but there is no quiver-split sub-folder.

Any help would be appreciated.

[NicMAlexandre]

The output ends with the following:
Exception: Some tasks are recently_done but not satisfied: set([Node(4-polish/quiver-run/000048F_004)])

Based on this stderr file, it looks like this is the step that failed

/global/scratch/nalexandre/spikedace/4-quiver/quiver-run/000048F_004

When I look at this sequence in the 3-unzip/all_h_ctg.fa file, it is not of zero-length, so not sure what's going on.