pacificbiosciences / pbbioconda Goto Github PK

PacBio Secondary Analysis Tools on Bioconda. Contains list of PacBio packages available via conda.

License: BSD 3-Clause Clear License

pbbioconda's Introduction

PacBio Secondary Analysis Tools on Bioconda

Information

PacBio® tools distributed via Bioconda are: pre-release versions, not necessarily ISO compliant, intended for Research Use Only and not for use in diagnostic procedures, intended only for command-line users, and possibly newer than the currently available SMRT® Analysis builds. While efforts have been made to ensure that releases on Bioconda live up to the quality that PacBio strives for, we make no warranty regarding any Bioconda release.

Support

As PacBio tools distributed via Bioconda are not covered by any service level agreement or the like, please do not contact a PacBio Field Applications Scientist or PacBio Customer Service for assistance with any Bioconda release. We instead provide an issue tracker for you to report issues to us at: https://github.com/PacificBiosciences/pbbioconda. We make no warranty that any such issue will be addressed, to any extent or within any time frame.

Official tech support is only provided for official and stable SMRT Link releases provided by PacBio.

No support via mail to developers.

For issues with the installation of conda or adding the bioconda channel, please refer to the official bioconda website.

For issues with installed PacBio binaries through bioconda, please create a new issue! Our developers will triage your problems internally and reply as soon as possible.

File sharing

If required to share a file for easier troubleshooting, please use PacBio's file drop-off. You do not need a "request code". Enter your name and mail to receive a drop-off link. Once received via mail, you will be forwarded to your personal drop-off page, where you will be asked to enter a PacBio recipient. Please enter [email protected] and proceed to upload your files. More info about file transfer security.

Installation

Information how to install conda and add the bioconda channel is available on https://bioconda.github.io/. Please verify that you have set up conda channel priorities according to set up conda channels.

Packages can be installed using following command:

> conda install -c bioconda package_name

Packages can be updated with:

> conda update package_name

In general, because conda only performs the smallest set of updates in order to satisfy the dependency graph, it is strongly advised to always update the environment as a whole afterwards:

> conda update --all

In order to avoid stale dependencies in the dependency graph.

Availability

Notes:

Python packages require version 2.7; Python 3 is currently not supported! You cannot install PacBio packages alongside python 3-only packages in the same environment! In order to use python 3-only packages, create a separate conda env.
BAM refers to the PacBio BAM format that includes additional per-record or per-ZMW information
Packages might conflict with other bioconda packages not maintained by PacBio

Packages

Package	Linux	Mac	Description
isoseq	Y	-	Scalable de novo isoform discovery (Release Notes)
jasmine	Y	-	Predict 5mC in PacBio HiFi reads (Release Notes)
lima	Y	-	Demultiplex barcoded samples (Release Notes)
pbaa	Y	Y	HiFi specific Amplicon Analysis
pbbam	Y	Y	BAM C++ library, tools are now in pbtk (Release Notes)
pbccs	Y	-	Generate HiFi reads for Sequel II (Release Notes)
pbcommand	Y	Y	Common models, CLI tool contract and SMRT Link service interface
pbcopper	Y	Y	Core C++ library for data structures, algorithms, and utilities (Release Notes)
pbcore	Y	Y	Python library for reading and writing data files
pbcoretools	Y	Y	CLI tools and add-ons for PacBio's core APIs (incl `dataset`)
pbfusion	Y	-	A fusion gene detection tool for all Iso-Seq data types (Release Notes)
pbipa	Y	-	Construct very contiguous, high quality de novo genome assemblies using the IPA HiFi assembler. (Release Notes)
pbmarkdup	Y	-	Mark duplicate reads from PacBio sequencing of an amplified library (Release Notes)
pbmm2	Y	-	A minimap2 frontend for PacBio native data formats (Release Notes)
pbpigeon	Y	-	PacBio transcript toolkit (Release Notes)
pbskera	Y	-	Read deconcatination (Release Notes)
pbsv	Y	-	Structural variant analysis (Release Notes)
pbtk	Y	-	PacBio BAM toolkit contains `bam2fasta`, `bam2fastq`, `ccs-kinetics-bystrandify`, `extracthifi`, `pbindex`, `pbindexdump`, `pbmerge`, `zmwfilter` (Release Notes)
recalladapters	Y	-	Recall adapters (Release Notes)
trgt	Y	-	Tandem repeat genotyping and visualization (Release Notes)

Combo-Packages

These combine multiple repos into a single bioconda package.

Package	Linux	Mac	Description
pb-falcon	Y	-	pypeflow/FALCON/FALCON_unzip (Release Notes)

Available Meta-Packages

These include dependencies only. They describe a mutually consistent, well-tested set of versions of all dependencies.

Package	Linux	Mac	Description
pb-assembly	Y	-	Everything needed to run Falcon and Unzip (Release Notes)

FAQ

Where do I get support?

PacBio tools distributed via bioconda are not covered by any service level agreement or the like. As such, please do not contact a PacBio Field Applications Scientist or PacBio Customer Service for assistance with any bioconda releases. We instead provide an issue tracker for you to report issues.

I can't find tool X, when will it be available on bioconda?

We do not provide ETAs for currently not available tools.

When will a new version of tool X be available?

We do not provide ETAs for our release schedule.

Will SMRT Link be available on bioconda?

There are no plans. For the latest SMRT Link release, please refer to currently available SMRT Link downloads.

Can I get the exact version of the binaries from SMRT Link version XY through bioconda?

There is no effort to keep official SMRT Link releases and bioconda binaries in sync.

Which version of tool X shall I use, from bioconda or from SMRT Link?

If you need ISO-compliant software that has been fully vetted, you must use our official SMRT Link software. Bioconda binaries are pre-release versions that adhere to high standards, but might generate slightly different output, due to bug fixes and/or new features.

Why is the official SMRT Tools Guide not in sync with bioconda binaries?

Official documentation provided by PacBio is for binaries distributed in stable SMRT Link releases. You can click on package names in the table above to get to unofficial, best effort documentation. Combo- and meta-packages don't necessarily have additional documentation, as they serve as lightweight, yet well-tested, sets of existing packages.

Which operating systems are supported?

All packages are available for 64-bit linux. Some are available for 64-bit MacOS. For details, please study the table above. There are no plans to provide darwin binaries for packages currently missing MacOS. There are no plans to provide executables for Windows. We do not provide support for WSL (Windows Subsystem for Linux).

Disclaimer/Copyright

© Copyright Pacific Biosciences of California, Inc. All rights reserved. Pacific Biosciences, the Pacific Biosciences logo, PacBio, SMRT, SMRTbell, Iso-Seq and Sequel are trademarks of Pacific Biosciences. All other trademarks are the sole property of their respective owners. Certain notices, terms, conditions and/or use restrictions may pertain to your use of Pacific Biosciences products and/or third party products. Please refer to the applicable Pacific Biosciences Terms and Conditions of Sale and to the applicable license terms at http://www.pacb.com/legal-and-trademarks/product-license-and-use-restrictions/. Information herein is subject to change without notice. Pacific Biosciences assumes no responsibility for any errors or omissions herein.

THIS WEBSITE AND CONTENT AND ALL SITE-RELATED SERVICES, INCLUDING ANY DATA, ARE PROVIDED "AS IS," WITH ALL FAULTS, WITH NO REPRESENTATIONS OR WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTIES OF MERCHANTABILITY, SATISFACTORY QUALITY, NON-INFRINGEMENT OR FITNESS FOR A PARTICULAR PURPOSE. YOU ASSUME TOTAL RESPONSIBILITY AND RISK FOR YOUR USE OF THIS SITE, ALL SITE-RELATED SERVICES, AND ANY THIRD PARTY WEBSITES OR APPLICATIONS. NO ORAL OR WRITTEN INFORMATION OR ADVICE SHALL CREATE A WARRANTY OF ANY KIND. ANY REFERENCES TO SPECIFIC PRODUCTS OR SERVICES ON THE WEBSITES DO NOT CONSTITUTE OR IMPLY A RECOMMENDATION OR ENDORSEMENT BY PACIFIC BIOSCIENCES.

pbbioconda's People

Contributors

Stargazers

Watchers

pbbioconda's Issues

Python3 compatibility

"we have a broken Arrow"

Operating system
Rehat 6.9

Package name
Arrow Via conda install (within its own env)

conda update pbcoretools

PackageNotInstalledError: Package is not installed in prefix.
prefix: /shelf/apps/pjt6/conda/envs/pb_toolkit
package name: pbcoretools

Conda environment
What is the result of conda list? (Try to paste that between triple backticks.)

#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                py27_1003    conda-forge
avro-python2              1.8.2                      py_1    bioconda
bcftools                  1.9                  h4da6232_0    bioconda
bedtools                  2.27.1               he941832_2    bioconda
blas                      1.0                         mkl
blasr                     5.3.2                hac9d22c_3    bioconda
blasr_libcpp              5.3.1                hac9d22c_2    bioconda
bwa                       0.7.17               ha92aebf_3    bioconda
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.10.15           ha4d7672_0    conda-forge
certifi                   2018.10.15            py27_1000    conda-forge
cffi                      1.11.5           py27h5e8e0c9_1    conda-forge
chardet                   3.0.4                 py27_1003    conda-forge
cryptography              2.3.1            py27hdffb7b8_0    conda-forge
cryptography-vectors      2.3.1                 py27_1000    conda-forge
curl                      7.61.0               h93b3f91_2    conda-forge
cython                    0.28.5           py27hfc679d8_0    conda-forge
decorator                 4.3.0                      py_0    conda-forge
enum34                    1.1.6                 py27_1001    conda-forge
future                    0.16.0                py27_1002    conda-forge
genomicconsensus          2.3.2                    py27_2    bioconda
h5py                      2.8.0            py27h7eb728f_3    conda-forge
hdf5                      1.10.2               hc401514_2    conda-forge
htslib                    1.7                           0    bioconda
idna                      2.7                   py27_1002    conda-forge
intel-openmp              2019.0                      118
ipaddress                 1.0.22                     py_1    conda-forge
iso8601                   0.1.12                     py_1    conda-forge
krb5                      1.14.6                        0    conda-forge
libdeflate                1.0                  h470a237_0    bioconda
libffi                    3.2.1                hfc679d8_5    conda-forge
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 7.2.0                hdf63c60_3    conda-forge
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libssh2                   1.8.0                h5b517e9_2    conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3    conda-forge
linecache2                1.0.0                      py_1    conda-forge
minimap2                  2.12                 ha92aebf_0    bioconda
mkl                       2019.0                      118
mkl_fft                   1.0.6                    py27_0    conda-forge
mkl_random                1.0.1                    py27_0    conda-forge
mummer4                   4.0.0beta2      pl526hfc679d8_3    bioconda
ncurses                   6.1                  hfc679d8_1    conda-forge
networkx                  2.2                        py_1    conda-forge
nim-falcon                0.0.0                         0    bioconda
numpy                     1.15.0           py27h1b885b7_0
numpy-base                1.15.0           py27h3dfced4_0
openjdk                   8.0.152              h46b5887_1
openssl                   1.0.2p               h470a237_1    conda-forge
pb-assembly               0.0.1                    py27_0    bioconda
pb-dazzler                0.0.0                h470a237_0    bioconda
pb-falcon                 0.2.3            py27ha92aebf_0    bioconda
pbalign                   0.3.1                    py27_0    bioconda
pbbam                     0.18.0               h1310cd9_1    bioconda
pbcommand                 1.1.1                    py27_2    bioconda
pbcore                    1.6.5                    py27_0    bioconda
perl                      5.26.2               h470a237_0    conda-forge
pilon                     1.22                          1    bioconda
pip                       18.1                  py27_1000    conda-forge
pycparser                 2.19                       py_0    conda-forge
pyopenssl                 18.0.0                   py27_0    conda-forge
pysam                     0.14.1           py27hae42fb6_1    bioconda
pysocks                   1.6.8                 py27_1002    conda-forge
python                    2.7.15               h33da82c_1    conda-forge
python-consensuscore      1.1.1            py27h02d93b8_2    bioconda
python-consensuscore2     3.1.0                    py27_2    bioconda
python-edlib              1.2.3            py27h470a237_1    bioconda
python-intervaltree       2.1.0                      py_0    bioconda
python-msgpack            0.5.6            py27h470a237_0    bioconda
python-sortedcontainers   2.0.4                      py_0    bioconda
pytz                      2018.5                     py_0    conda-forge
readline                  7.0                  haf1bffa_1    conda-forge
requests                  2.19.1                   py27_1    conda-forge
samtools                  1.9                  h8ee4bcc_1    bioconda
setuptools                40.4.3                   py27_0    conda-forge
six                       1.11.0                py27_1001    conda-forge
sqlite                    3.25.2               hb1c47c0_0    conda-forge
tk                        8.6.8                ha92aebf_0    conda-forge
traceback2                1.4.0                    py27_0    conda-forge
unittest2                 1.1.0                      py_0    conda-forge
urllib3                   1.23                     py27_1    conda-forge
wheel                     0.32.1                   py27_0    conda-forge
xz                        5.2.4                h470a237_1    conda-forge
zlib                      1.2.11               h470a237_3    conda-forge```


**Describe the bug**
when trying to use Arrow, I always get the following bug. Arrow continues to "do something", but never outputs any data to the gff, fasta or fastq file. It has been running for 7 hours. All files are still empty. Previosu version of Arrow put data out as it was working. 

**Error message**
> arrow -h
/shelf/apps/pjt6/conda/envs/pb_toolkit/lib/python2.7/site-packages/h5py/__init__.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  from ._conv import register_converters as _register_converters
/shelf/apps/pjt6/conda/envs/pb_toolkit/lib/python2.7/site-packages/h5py/__init__.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  from . import h5a, h5d, h5ds, h5f, h5fd, h5g, h5r, h5s, h5t, h5p, h5z
/shelf/apps/pjt6/conda/envs/pb_toolkit/lib/python2.7/site-packages/h5py/_hl/group.py:22: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  from .. import h5g, h5i, h5o, h5r, h5t, h5l, h5p

**To Reproduce**
conda create -n pb_toolkit pbcoretools
conda update pbcoretools
conda activate pb_toolkit


THREADS=16
PREFIX=species_name
###################################################
# Iter 1
ITERNUM=1
minimap2 -t $THREADS -ax map-pb Assembly_unpolished.fasta PacBio.fastq.gz > aln.sam
samtools view -@ $THREADS -S -b -o alnunsorted.bam aln.sam
wait
samtools sort -@ $THREADS -o  sorted_mini2.bam alnunsorted.bam
samtools index sorted_mini2.bam
pbbamify --input=sorted_mini2.bam --output=sorted_mini.unsorted.pb.bam Gpal_newton_final_unpolished.fasta mydataset.xml
samtools sort -@ $THREADS -o sorted_mini.pb.bam sorted_mini.unsorted.pb.bam 
pbindex  sorted_mini.pb.bam

arrow  sorted_mini.pb.bam  --diploid --log-file miniarrow.log -j $THREADS --referenceFilename Assembly_unpolished.fasta -o ${PREFIX}.arrow${ITERNUM}.fasta -o ${PREFIX}.arrow${ITERNUM}.gff -o ${PREFIX}.arrow${ITERNUM}.fastq



**Expected behavior**
Arrow to ouput data into the fasta, gff and fastq files as it goes through the bam file. No output after 7 hours (based on previous experience of Arrow - this is not right). 
This is unexpected: binary incompatibility. Expected 96, got 88

Trouble in "graph annotation and haplotig" step in FALCON-Unzip

We are working on a plant with 2.2 Gb genome. The FALCON assembly was completed with 107 Gb Sequel reads (~50x), but FALCON-Unzip failed at the "graph annotation and haplotig" step of a contig, saying:

Exception: Call 'nucmer -mum p_ctg.002484F.fa h_ctg_all.002484F.fa -p hp_aln' returned 256.

It looks abortion in nucmer process because the primary contig (p_ctg) size in 1-hasm/ was 0.

The FALCON-Unzip always failed at this contig even we tried again and again.

How should we fix it? Or, can we skip this contig?

download link for model.json needed for unanimity

The page https://github.com/PacificBiosciences/unanimity describes how to place a file named model.json in a given location

In such cases, download the latest version of the model parameters and place them in a subdirectory of ${SMRT_CHEMISTRY_BUNDLE_DIR}:

but I find nowhere a link to download that file and do not find it in my SMRTLink tree either.

PB-Assembly

Operating system
Linux, CentOS7

Package name
falcon-kit 1.2.3
pypeflow 2.1.0

Describe the issue
I am running the falcon job in our Moab PBS system, so I don’t have the -W block=T option available. And given this post:adaptivecomputing/torque#268, I don’t think Moab will implement some blocking function in the near future.

So instead of using “pwatcher_type=blocking”, I used “pwatcher_type=fs_based”.

It turned out this option is not well implemented, for example, for some of the tasks, it worked, but for others, it failed.
When it fails, the error message will (always) be something like:
[INFO]CALL:
qdel Pedec92ba939bd0
qdel: illegally formed job identifier: Pedec92ba939bd0

I suspect this means falcon tries to kill an already-killed job after it detects the "run.sh.done" file. Normally A simply re-submission of the same job script will resume the whole pipeline. But I am just wondering whether we could fix this issue so that people don't need to re-submit the same job over and over.

Then I tried to use a combination of “-I -x” in hope to make an equivalent case for -W block=T (see this post: https://stackoverflow.com/questions/5982857/making-qsub-block-until-job-is-done). This time, the job keeps running, however, even the pbs job is “done” (status C in queue), there is no “run.sh.done” file generated in the designated directory. I am not sure whether this is the default behavior, or I hit another bug.

Error message
when using "pwatcher_type=fs_based":
[INFO]CALL:
qdel Pedec92ba939bd0
qdel: illegally formed job identifier: Pedec92ba939bd0

pbsv Ignores Signals

pbsv does not respond to Ctrl-C (SIGINT). To stop it, I have to SIGKILL. This would make it impossible to stop gracefully in a distributed system with 100's of nodes.

I am currently testing alignments with pbsv fasta, minimap2, and samtools (see rule below). When terminated (Ctrl-C), everything shuts down but the pbsv process. Could pbsv just exit when signaled?

Command:

        shell((
            """pbsv fasta {INPUT_FILE} | """
            """minimap2 -x map-pb -a --eqx -L -O 5,56 -E 4,1 -B 5 --secondary=no -z 400,50 -r 2k -Y -R"""
                """ "@RG\\tID:rg{SAMPLE}-{MOVIE}\\tSM:{SAMPLE}" {REF_FA} - | """
            """samtools sort -T {TEMP_DIR}/sort_ > """
            """{OUTPUT_FILE}; """
            """rm -rf {TEMP_DIR} """
        ).format(**param_dict))

Operating system
CentOS 6 (2.6.32-696.18.7.el6.x86_64)

Package name
pbsv

Conda environment

# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py27_3    conda-forge
avro-python2              1.8.2                      py_1    bioconda
bam2fastx                 1.3.0                h4ef8376_5    bioconda
bax2bam                   0.0.9                hac9d22c_4    bioconda
bcftools                  1.9                  h4da6232_0    bioconda
bedtools                  2.27.1               he941832_2    bioconda
blas                      1.0                         mkl  
blasr                     5.3.2                hac9d22c_4    bioconda
blasr_libcpp              5.3.1                hac9d22c_3    bioconda
bwa                       0.7.17               ha92aebf_3    bioconda
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.8.24            ha4d7672_0    conda-forge
certifi                   2018.8.24             py27_1001    conda-forge
cffi                      1.11.5           py27h5e8e0c9_1    conda-forge
chardet                   3.0.4                    py27_3    conda-forge
cryptography              2.3.1            py27hdffb7b8_0    conda-forge
cryptography-vectors      2.3.1                    py27_0    conda-forge
curl                      7.61.0               h93b3f91_2    conda-forge
cython                    0.28.5           py27hfc679d8_0    conda-forge
decorator                 4.3.0                      py_0    conda-forge
enum34                    1.1.6                 py27_1001    conda-forge
future                    0.16.0                   py27_2    conda-forge
genomicconsensus          2.3.2                    py27_2    bioconda
h5py                      2.8.0            py27hb794570_1    conda-forge
hdf5                      1.10.2               hc401514_2    conda-forge
htslib                    1.9                  hc238db4_4    bioconda
idna                      2.7                      py27_2    conda-forge
intel-openmp              2019.0                      118  
ipaddress                 1.0.22                     py_1    conda-forge
iso8601                   0.1.12                     py_1    conda-forge
isoseq3                   3.0.0                         2    bioconda
krb5                      1.14.6                        0    conda-forge
libdeflate                1.0                  h470a237_0    bioconda
libffi                    3.2.1                hfc679d8_5    conda-forge
libgcc-ng                 7.2.0                hdf63c60_3    conda-forge
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libssh2                   1.8.0                h5b517e9_2    conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3    conda-forge
lima                      1.8.0                         0    bioconda
linecache2                1.0.0                    py27_0    conda-forge
minimap2                  2.12                 ha92aebf_0    bioconda
minorseq                  1.11.0                        1    bioconda
mkl                       2019.0                      118  
mkl_fft                   1.0.6                    py27_0    conda-forge
mkl_random                1.0.1                    py27_0    conda-forge
mummer4                   4.0.0beta2      pl526hfc679d8_3    bioconda
ncurses                   6.1                  hfc679d8_1    conda-forge
networkx                  2.2                        py_1    conda-forge
nim-falcon                0.0.0                         0    bioconda
numpy                     1.15.0           py27h1b885b7_0  
numpy-base                1.15.0           py27h3dfced4_0  
openblas                  0.2.20                        8    conda-forge
openssl                   1.0.2p               h470a237_0    conda-forge
pb-assembly               0.0.0                    py27_8    bioconda
pb-dazzler                0.0.0                h470a237_0    bioconda
pb-falcon                 0.2.1            py27hf50d5a6_0    bioconda
pbalign                   0.3.1                    py27_0    bioconda
pbbam                     0.19.0               h6678c95_0    bioconda
pbccs                     3.1.0                         3    bioconda
pbcommand                 1.1.1                    py27_2    bioconda
pbcopper                  0.4.2                h02d93b8_0    bioconda
pbcore                    1.5.1                    py27_2    bioconda
pbcoretools               0.2.4                    py27_2    bioconda
pblaa                     2.4.2                         1    bioconda
pbmm2                     0.9.0                ha87ae23_0    bioconda
pbsv                      2.0.1                         0    bioconda
perl                      5.26.2               h470a237_0    conda-forge
pip                       18.0                     py27_1    conda-forge
pycparser                 2.19                       py_0    conda-forge
pyopenssl                 18.0.0                   py27_0    conda-forge
pysam                     0.15.1           py27h0380709_0    bioconda
pysocks                   1.6.8                    py27_2    conda-forge
python                    2.7.15               h9fef7bc_0    conda-forge
python-consensuscore      1.1.1            py27h02d93b8_2    bioconda
python-consensuscore2     3.1.0                    py27_2    bioconda
python-edlib              1.2.3            py27h470a237_1    bioconda
python-intervaltree       2.1.0                      py_0    bioconda
python-msgpack            0.5.6            py27h470a237_0    bioconda
python-sortedcontainers   2.0.4                      py_0    bioconda
pytz                      2018.5                     py_0    conda-forge
readline                  7.0                  haf1bffa_1    conda-forge
requests                  2.19.1                   py27_1    conda-forge
samtools                  1.9                  h8ee4bcc_1    bioconda
setuptools                40.4.0                py27_1000    conda-forge
six                       1.11.0                   py27_1    conda-forge
sqlite                    3.25.1               hb1c47c0_0    conda-forge
tk                        8.6.8                ha92aebf_0    conda-forge
traceback2                1.4.0                    py27_0    conda-forge
unittest2                 1.1.0                      py_0    conda-forge
urllib3                   1.23                     py27_1    conda-forge
wheel                     0.31.1                py27_1001    conda-forge
xz                        5.2.4                h470a237_1    conda-forge
zlib                      1.2.11               h470a237_3    conda-forge

BLASR error

Operating system
CentOS 7 64 bit

Package name
BLASR version 5.3.2.

[vk@GenomeBiology work]$ python ~/distr/halc/runHALC.py d_PB.fa short_reads_A.fasta -t 22 -r
2018-11-02 16:26:58.353467

/////STEP 1 STARTED//////////////////////////////////////////////////////////////////////////////////////////////////
Running command: Chunker -s 200M -o ./temp/step1/pb-%03d.fa d_PB.fa 1>./temp/step1/SeqChunker.out 2>./temp/step1/SeqChunker.err
25 files created

/////STEP 1 DONE/////////////////////////////////////////////////////////////////////////////////////////////////////

/////STEP 2 STARTED//////////////////////////////////////////////////////////////////////////////////////////////////
Running command: blasr ./temp/step1/pb-001.fa short_reads_A.fasta -m 5 --out ./temp/step2/blasrresult-001.m5 --maxScore 2000 --minMatch 8 --minAlnLength 300 --nCandidates 30 --bestn 20 --nproc 22 1>./temp/step2/blasr_1.out 2>./temp/step2/blasr_1.err
ERROR: Failed to run BLASR: Unknown error 256

[vk@GenomeBiology work]$ more temp/step2/blasr_1.out 
ERROR! Reading fasta files greater than 4Gbytes is not supported.

-rw-rw-r--. 1 vk vk 203G ноя 2 15:48 short_reads_A.fasta
-rw-rw-r--. 1 vk vk 4,9G окт 26 12:53 d_uni_PB.fa

pb-assembly needs a mechanism for the user to control the environment when any task is executed.

Operating system
Redhat Enterprise 6, but the issue has nothing to do with the OS.

Package name
pb-assembly

Describe the bug
The pypeflow process depends upon having the local environment passed to any processes that execute tasks. However, under the Univa Grid Engine (a flavor of SGE), it is possible to configure the system so no environment variables are passed to submitted jobs. Thus, no tasks can work.

Error message
Within an stderr from a qsub command, you can get an error like:
+ /bin/bash task.sh
/db/congenomics/local6/binaries/python-2.7.10/bin/python2.7: No module named pypeflow

To Reproduce
Run an analysis on an SGE system that doesn't pass the environment when -V is specified in the qsub command.

Expected behavior
N/A

What's need is an additional configuration option for the job.defaults stanza that allows the user to specify a command to be executed at the very beginning of any task.sh file. As an example, in my environment,

export PATH=/db/congenomics/local6/binaries/conda/bin:$PATH

In pb-assembly, the following locations are candidates for where the above command should be executed:

./lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py:437:set -vex
./lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py:510:        script_fn = os.path.join(wdir, 'task.sh')
./lib/python2.7/site-packages/falcon_unzip/tasks/unzip.py:254:set -vex
./lib/python2.7/site-packages/falcon_kit/bash.py:17:    ofs.write('set -vex\n')
./lib/python2.7/site-packages/falcon_kit/snakemake.py:227:set -vex
./lib/python2.7/site-packages/pwatcher/mains/pypeflow_example.py:44:set -vex
./lib/python2.7/site-packages/pwatcher/mains/pypeflow_example.py:70:set -vex

bioconda package numpy dependency version

I'm unclear if this should reported here, or on the BioConda repository (where I could submit a pull request to update the meta.yaml file).

Operating system
Linux

Package name
Which package / tool is causing the problem? Which version are you using, use tool --version. Have you updated to the latest version conda update package?

Using genomicconsensus-2.3.2 (latest version), bug shown with variantCaller --version as described below.

Describe the bug
A clear and concise description of what the bug is.

API incompatibility warning with genomicconsensus-2.3.2 and numpy-1.15.1

Error message
Paste the error message / stack.

$ variantCaller --version
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
2.3.2

To Reproduce
Steps to reproduce the behavior. Providing a minimal test dataset on which we can reproduce the behavior will generally lead to quicker turnaround time!

$ conda install numpy==1.13.3
...
$ conda install genomicconsensus
...
Downloading and Extracting Packages
python-consensuscore | 848 KB    | #################################################################################################################### | 100% 
python-consensuscore | 1.4 MB    | #################################################################################################################### | 100% 
genomicconsensus-2.3 | 96 KB     | #################################################################################################################### | 100% 
pbcommand-1.1.1      | 193 KB    | #################################################################################################################### | 100% 
pbcore-1.5.1         | 9.7 MB    | #################################################################################################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction:
...

Then,

$ variantCaller --version
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
2.3.2
$ arrow --version
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
2.3.2

Updating numpy fixed this,

$ conda update numpy
...
The following packages will be UPDATED:

    blas:        1.0-mkl                           --> 1.1-openblas                         conda-forge
    dextractor:  1.0p2-0               bioconda    --> 1.0p2-h647bee3_1                     bioconda   
    h5py:        2.7.0-np113py27_0                 --> 2.8.0-py27hb794570_1                 conda-forge
    hdf5:        1.8.17-11             conda-forge --> 1.10.2-hc401514_2                    conda-forge
    numpy:       1.13.3-py27hdbf6ddf_4             --> 1.15.1-py27_blas_openblashd3ea46f_1  conda-forge [blas_openblas]
    numpy-base:  1.14.3-py27hdbf6ddf_0             --> 1.14.3-py27h0ea5e3f_1                           
    scipy:       1.1.0-py27hfc37229_0              --> 1.1.0-py27_blas_openblash7943236_201 conda-forge [blas_openblas]

Proceed ([y]/n)? y


Downloading and Extracting Packages
hdf5-1.10.2          | 4.8 MB    | #################################################################################################################### | 100% 
linecache2-1.0.0     | 22 KB     | #################################################################################################################### | 100% 
libopenblas-0.2.20   | 8.8 MB    | #################################################################################################################### | 100% 
numpy-base-1.14.3    | 4.0 MB    | #################################################################################################################### | 100% 
dextractor-1.0p2     | 100 KB    | #################################################################################################################### | 100% 
unittest2-1.1.0      | 68 KB     | #################################################################################################################### | 100% 
traceback2-1.4.0     | 28 KB     | #################################################################################################################### | 100% 
numpy-1.15.1         | 8.9 MB    | #################################################################################################################### | 100% 
scipy-1.1.0          | 39.4 MB   | #################################################################################################################### | 100% 
h5py-2.8.0           | 3.4 MB    | #################################################################################################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
[pc40583@n13-16-384-hypnotoad assembly]$ 
[pc40583@n13-16-384-hypnotoad assembly]$ 
[pc40583@n13-16-384-hypnotoad assembly]$ arrow --version
2.3.2

Expected behavior
A clear and concise description of what you expected to happen.

The dependencies Arrow (etc) should ensure a version of NumPy compatible with what was used to compile the binaries gets installed on conda.

From trying $ conda install numpy==1.14 which also works, it seems we need at least numpy 1.14 in the recipe's meta.yaml

About the test data

about the test data:10-fold coverage of the Genome in a Bottle sample HG002
i call the vcf from the given bam is quite different with that vcf you given(https://downloads.pacbcloud.com/public/dataset/HG002/Sequel-201810/GRCh38.HG002_pbsv.vcf)
i use reference genome GCA_000001405.15_GRCh38_no_alt_analysis_set, do you use the default parameter, or use repeat region with -b.
Thanks!

pbalign maximum memory per thread hard coded to 4G

Hi,
During the pbalign samtools sort step, there are two lines where the memory per thread is hard coded to 4G. Can this be updated to allow the user to control this value?
Package name
pbalign version 0.3.1
.conda/envs/arrow/lib/python2.7/site-packages/pbalign/bampostservice.py

        if _stvmajor >= 1:
            cmd = 'samtools sort --threads {t} -m 4G -o {sortedBamFile} {unsortedBamFile}'.format(
                t=nproc, sortedBamFile=sortedBamFile, unsortedBamFile=unsortedBamFile)
        else:
            cmd = 'samtools sort --threads {t} -m 4G {unsortedBamFile} {prefix}'.format(
                t=nproc, unsortedBamFile=unsortedBamFile, prefix=sortedPrefix)
        Execute(self.name, cmd)

missing libpbcopper

Operating system
Which operating system and version are you using?
CentOS 6.10

Package name
Which package / tool is causing the problem? Which version are you using, use tool --version. Have you updated to the latest version conda update package?
pb-assembly 0.0.2-py27_0 bioconda
Conda environment
What is the result of conda list? (Try to paste that between triple backticks.)

Describe the bug
A clear and concise description of what the bug is.
I installed pb-assembly using bioconda and was able to run pb-falcon but pb-unzip fails at 4-polish/cns-output step. After investigating further it appears the error actually happened at quiver-run phase where variantCaller failed due to missing libpbcopper.so.4.2, error pasted below.

Error message
Paste the error message / stack.
variantCaller --algorithm=arrow -x 5 -X 120 -q 20 -j 24 -r ../../../quiver-split/refs/000000F/ref.fa aln-000000F.bam -o cns.fasta.gz -o cns.fastq.gz --minConfidence 0 -o cns.vcf
libpbcopper.so.0.4.2: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "/scicomp/home/bun3/.local/lib/python2.7/site-packages/pbcommand/cli/core.py", line 138, in _pacbio_main_runner
return_code = exe_main_func(*args, **kwargs)
File "/scicomp/home/bun3/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/main.py", line 340, in args_runner
return tr.main()
File "/scicomp/home/bun3/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/main.py", line 269, in main
self._algorithm = self._algorithmByName(options.algorithm, peekFile)
File "/scicomp/home/bun3/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/main.py", line 57, in _algorithmByName
from GenomicConsensus.arrow import arrow
File "/scicomp/home/bun3/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/arrow/init.py", line 4, in
from . import utils
File "/scicomp/home/bun3/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/arrow/utils.py", line 11, in
import ConsensusCore2 as cc
File "/scicomp/home/bun3/.local/lib/python2.7/site-packages/ConsensusCore2.py", line 17, in
_ConsensusCore2 = swig_import_helper()
File "/scicomp/home/bun3/.local/lib/python2.7/site-packages/ConsensusCore2.py", line 16, in swig_import_helper
return importlib.import_module('_ConsensusCore2')
File "/scicomp/groups/OID/NCEZID/DSR/BCFB/usr/bin/miniconda2/denovo_asm/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
ImportError: libpbcopper.so.0.4.2: cannot open shared object file: No such file or directory

To Reproduce
Steps to reproduce the behavior. Providing a minimal test dataset on which we can reproduce the behavior will generally lead to quicker turnaround time!
This was run on the Exanple Test data greg200k-sv2

Expected behavior
A clear and concise description of what you expected to happen.
I expect the test data to finish running

All tasks need to be restartable to deal with job queueing problems

I'm currently working on a very large genome (>10 Gbp), and the Linux cluster (which uses a variant of SGE) that I'm using is not 100% reliable (not unexpected given the size of the cluster). It is frequently the case that jobs will need to be restarted in order to deal with problems on nodes.

In addition, we have job submission policies that require users to specify the maximum elapsed time for jobs, and there are significant throughput penalties for overestimating the required time. Thus, there is a strong incentive to submit jobs with a modest time expectation and resubmit the job with a longer time limit if the time limit is exceeded.

Thus, it is essential in our environment to have all tasks be restartable. In other words, a task should not assume that its directory are empty, but rather should expect that partial results may exist, and should be removed or ignored if the previous invocation of the task had failed.

ERROR - Was waiting for '0-rawreads/tan-split/tan-uows.json'

Hi,

Could anyone please give an insight to why I am getting the following error?

2018-10-16 06:58:34,551 - root:74 - DEBUG - Checking existence of '0-rawreads/tan-split/tan-uows.json' with timeout=30
2018-10-16 06:59:04,589 - root:70 - ERROR - Was waiting for '0-rawreads/tan-split/tan-uows.json'
Traceback (most recent call last):
File "/gpfs/software/genomics/pb-falcon/src/lib/python2.7/site-packages/pypeflow/do_task.py", line 68, in wait_for
_wait_for(fn, timeout)
File "/gpfs/software/genomics/pb-falcon/src/lib/python2.7/site-packages/pypeflow/do_task.py", line 84, in _wait_for
raise Exception('Timed out waiting for {!r}'.format(fn))
Exception: Timed out waiting for '0-rawreads/tan-split/tan-uows.json'

My config file as as follows:

#### Input
[General]
input_fofn=input.fofn
input_type=raw
pa_DBdust_option=
pa_fasta_filter_option=streamed-median
target=assembly
skip_checks=False
LA4Falcon_preload=false

#### Data Partitioning
pa_DBsplit_option=-x500 -s400
ovlp_DBsplit_option=-s400

#### Repeat Masking
pa_HPCTANmask_option=
pa_REPmask_code=0,300;0,300;0,300

#### Pre-assembly
genome_size=3000000000
seed_coverage=50
length_cutoff=-1    
pa_HPCdaligner_option=-v -B128 -M24
pa_daligner_option= -k18 -e0.75 -l1200 -h256 -w8 -s100
falcon_sense_option=--output-multi --min-idt 0.70 --min-cov 4 --max-n-read 200
falcon_sense_greedy=False

#### Pread overlapping
ovlp_HPCdaligner_option=-v -B128 -M24 
ovlp_daligner_option=-k24 -e.92 -l1800 -h600 -s100

#### Final Assembly
length_cutoff_pr=1000
overlap_filtering_setting=--max-diff 100 --max-cov 100 --min-cov 2
fc_ovlp_to_graph_option=

[job.defaults]
job_type=lsf
pwatcher_type=blocking
JOB_QUEUE=fatnode
MB=999999
NPROC=20
njobs=240
submit = bsub -q fatnode -P fatnode -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} ${JOB_SCRIPT}

[job.step.da]
NPROC=20
MB=999999
njobs=240
[job.step.la]
NPROC=20
MB=999999
njobs=240
[job.step.cns]
NPROC=20
MB=999999
njobs=240
[job.step.pda]
NPROC=20
MB=999999
njobs=240
[job.step.pla]
NPROC=20
MB=999999
njobs=240
[job.step.asm]
NPROC=20
MB=999999
njobs=240

Error at the 4-quiver step of pb-assembly

Operating system
Linux 2.6.32-696.16.1.el6.x86_64 - Scientific Linux release 6.9 (Carbon)

Package name
I installed the last version of pb-assembly (version 0.0.0) with conda. I have a bug with fc_quiver.py script.

Conda environment

# packages in environment at /cm/shared/apps/FALCON_12_09_2018:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py27_0
avro                      1.8.0                    py27_0    bioconda
bcftools                  1.9                  h4da6232_0    bioconda
blas                      1.0                         mkl
blasr                     5.3.2                hac9d22c_3    bioconda
blasr_libcpp              5.3.1                hac9d22c_2    bioconda
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.8.24            ha4d7672_0    conda-forge
certifi                   2018.8.24                py27_1    conda-forge
cffi                      1.11.5           py27he75722e_1
chardet                   3.0.4                    py27_1
conda                     4.5.11                   py27_0    conda-forge
conda-env                 2.6.0                         1
cryptography              2.3.1            py27hc365091_0
curl                      7.61.0               h93b3f91_2    conda-forge
cython                    0.28.5           py27hfc679d8_0    conda-forge
decorator                 4.3.0                      py_0    conda-forge
enum34                    1.1.6                    py27_1
future                    0.16.0                   py27_2    conda-forge
futures                   3.2.0                    py27_0
genomicconsensus          2.3.2                    py27_1    bioconda
h5py                      2.8.0            py27hb794570_1    conda-forge
hdf5                      1.10.2               hc401514_2    conda-forge
htslib                    1.7                           0    bioconda
idna                      2.7                      py27_0
intel-openmp              2018.0.3                      0
ipaddress                 1.0.22                   py27_0
iso8601                   0.1.12                     py_1    conda-forge
krb5                      1.14.6                        0    conda-forge
libdeflate                1.0                  h470a237_0    bioconda
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 8.2.0                hdf63c60_1
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libssh2                   1.8.0                h5b517e9_2    conda-forge
libstdcxx-ng              8.2.0                hdf63c60_1
linecache2                1.0.0                    py27_0    conda-forge
minimap2                  2.12                 ha92aebf_0    bioconda
mkl                       2018.0.3                      1
mkl_fft                   1.0.6                    py27_0    conda-forge
mkl_random                1.0.1                    py27_0    conda-forge
mummer                    3.23                    pl526_6    bioconda
ncurses                   6.1                  hf484d3e_0
networkx                  2.1                        py_1    conda-forge
nim-falcon                0.0.0                         0    bioconda
numpy                     1.15.1           py27h1d66e8a_0
numpy-base                1.15.1           py27h81de0dd_0
openssl                   1.0.2p               h470a237_0    conda-forge
pb-assembly               0.0.0                    py27_3    bioconda
pb-dazzler                0.0.0                h470a237_0    bioconda
pb-falcon                 0.0.2                    py27_0    bioconda
pbalign                   0.3.1                    py27_0    bioconda
pbbam                     0.18.0               h1310cd9_1    bioconda
pbcommand                 1.1.1            py27h24bf2e0_1    bioconda
pbcore                    1.5.1                    py27_1    bioconda
perl                      5.26.2               h470a237_0    conda-forge
pip                       10.0.1                   py27_0
pycosat                   0.6.3            py27h14c3975_0
pycparser                 2.18                     py27_1
pyopenssl                 18.0.0                   py27_0
pysam                     0.14.1           py27hae42fb6_1    bioconda
pysocks                   1.6.8                    py27_0
python                    2.7.15               h1571d57_0
python-consensuscore      1.1.1            py27h02d93b8_1    bioconda
python-consensuscore2     3.1.0                    py27_1    bioconda
python-edlib              1.2.3            py27h470a237_1    bioconda
python-intervaltree       2.1.0                      py_0    bioconda
python-msgpack            0.5.6            py27h470a237_0    bioconda
python-sortedcontainers   2.0.4                      py_0    bioconda
pytz                      2018.5                     py_0    conda-forge
readline                  7.0                  h7b6447c_5
requests                  2.19.1                   py27_0
ruamel_yaml               0.15.46          py27h14c3975_0
samtools                  1.9                  h8ee4bcc_1    bioconda
setuptools                40.2.0                   py27_0
six                       1.11.0                   py27_1
sqlite                    3.24.0               h84994c4_0
tk                        8.6.8                hbc83047_0
traceback2                1.4.0                    py27_0    conda-forge
unittest2                 1.1.0                      py_0    conda-forge
urllib3                   1.23                     py27_0
wheel                     0.31.1                   py27_0
xz                        5.2.4                h470a237_1    conda-forge
yaml                      0.1.7                had09818_2
zlib                      1.2.11               ha838bed_2

Describe the bug

The step "3-unzip" of pb-assembly works well, with the files "all_p_ctg.fa" and "all_h_ctg.fa" generated, but quiver step doesn't work. I have the impression that it is having an error at the end. Have you ever met this?

Here the folder 4-quiver. The last step that has worked seems to be quiver-run :

There are 2639 folders in quiver-run, so there are steps that have worked, but not all.

Error message

Here the log with the error :

 [INFO]Popen: 'bash -C /cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/pwatcher/mains/job_start.sh >| /data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001/run-P11ad2b39cd9e4b.bash
 .stdout 2>| /data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001/run-P11ad2b39cd9e4b.bash.stderr'
 [ERROR]Task Node(4-quiver/quiver-run/000153Fp01_00001) failed with exit-code=1
 [ERROR]Some tasks are recently_done but not satisfied: set([Node(4-quiver/quiver-run/000153Fp01_00001)])
 [ERROR]ready: set([Node(4-quiver/quiver-run/001238Fp01_00002), Node(4-quiver/quiver-run/000889Fp01), Node(4-quiver/quiver-run/000200Fp01), Node(4-quiver/quiver-run/002254Fp02), Node(4-quiver/quiver-run/001827
 Fp01), Node(4-quiver/quiver-run/000342Fp01), Node(4-quiver/quiver-run/000474Fp01), Node(4-quiver/quiver-run/000078Fp01_00001), Node(4-quiver/quiver-run/001592Fp01_00001), Node(4-quiver/quiver-run/001250Fp01),
  Node(4-quiver/quiver-run/000682Fp01), Node(4-quiver/quiver-run/001979Fp01), Node(4-quiver/quiver-run/001033Fp01_00001), Node(4-quiver/quiver-run/000477Fp01), Node(4-quiver/quiver-run/000495Fp01), Node(4-quiv
 er/quiver-run/002699Fp01), Node(4-quiver/quiver-run/000450Fp01_00001), Node(4-quiver/quiver-run/001441Fp01), Node(4-quiver/quiver-run/000209Fp01), Node(4-quiver/quiver-run/000584Fp01_00001), Node(4-quiver/qui
 ...................
 ver-run/000645Fp01), Node(4-quiver/quiver-run/000637Fp01), Node(4-quiver/quiver-run/000408Fp01_00003), Node(4-quiver/quiver-run/000176Fp01_00001), Node(4-quiver/quiver-run/002065Fp01), Node(4-quiver/quiver-ru
 n/000405Fp01_00002), Node(4-quiver/quiver-run/000675Fp01_00001), Node(4-quiver/quiver-run/000735Fp01_00001), Node(4-quiver/quiver-run/002196Fp02), Node(4-quiver/quiver-run/000743Fp01), Node(4-quiver/quiver-ru
 n/000559Fp01_00001), Node(4-quiver/quiver-run/000317Fp01_00002)])
         submitted: set([Node(4-quiver/quiver-run/000484Fp01_00001)])
 [ERROR]Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
 Traceback (most recent call last):
   File "/cm/shared/apps/FALCON_12_09_2018/bin//fc_quiver.py", line 11, in <module>
     load_entry_point('falcon-unzip==1.1.2', 'console_scripts', 'fc_quiver.py')()
   File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/falcon_unzip/mains/start_unzip.py", line 29, in main
     unzip.run(**vars(args))
   File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/falcon_unzip/unzip.py", line 126, in run
     unzip_all(config)
   File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/falcon_unzip/unzip.py", line 28, in unzip_all
     tasks_unzip.run_workflow(wf, config, rule_writer)
   File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/falcon_unzip/tasks/unzip.py", line 708, in run_workflow
    job_dict=config['job.step.unzip.quiver'],
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/falcon_kit/pype.py", line 192, in gen_parallel_tasks
     wf.refreshTargets()
   File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 277, in refreshTargets
     self._refreshTargets(updateFreq, exitOnFailure)
   File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 361, in _refreshTargets
     raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: set([Node(4-quiver/quiver-run/000153Fp01_00001)])
make: *** [quiver] Error 1

Here the "4-quiver/quiver-run/000153Fp01_00001/run-P11ad2b39cd9e4b.bash.stderr" file :

executable=${PYPEFLOW_JOB_START_SCRIPT}
+ executable=/data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001/run-P11ad2b39cd9e4b.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
+ timeout=60

# Wait up to timeout seconds for the executable to become "executable",
# then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
    if [[ "${timeout}" == "0" ]]; then
        echo "timed out waiting for (${executable})"
        exit 77
    fi
    echo "not executable: '${executable}', waiting ${timeout}s"
    sleep 1
    timeout=$((timeout-1))
done
+ [[ ! -x /data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001/run-P11ad2b39cd9e4b.bash ]]

/bin/bash ${executable}
+ /bin/bash /data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001/run-P11ad2b39cd9e4b.bash
+ '[' '!' -d /data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001 ']'
+ cd /data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001
+ eval '/bin/bash run.sh'
++ /bin/bash run.sh
export PATH=$PATH:/bin
+ export PATH=/cm/shared/apps/FALCON_12_09_2018/bin/:/cm/shared/apps/Perl_conda/bin:/cm/shared/apps/slurm/14.03.0/sbin:/cm/shared/apps/slurm/14.03.0/bin:/cm/local/apps/cluster-tools/bin:/cm/local/apps/cmd/sbi
n:/cm/local/apps/cmd/bin:/cm/shared/apps/cmgui:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/root/bin:/bin
+ PATH=/cm/shared/apps/FALCON_12_09_2018/bin/:/cm/shared/apps/Perl_conda/bin:/cm/shared/apps/slurm/14.03.0/sbin:/cm/shared/apps/slurm/14.03.0/bin:/cm/local/apps/cluster-tools/bin:/cm/local/apps/cmd/sbin:/cm/l
ocal/apps/cmd/bin:/cm/shared/apps/cmgui:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/root/bin:/bin
cd /data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001
+ cd /data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001
/bin/bash task.sh
+ /bin/bash task.sh
pypeflow 2.0.4
2018-09-25 01:34:24,412 - root - DEBUG - Running "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/pypeflow/do_task.py /data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001/task.js
on"
2018-09-25 01:34:24,414 - root - DEBUG - Checking existence of '/data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001/task.json' with timeout=30
2018-09-25 01:34:24,414 - root - DEBUG - Loading JSON from '/data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001/task.json'
2018-09-25 01:34:24,415 - root - DEBUG - {u'bash_template_fn': u'template.sh',
 u'inputs': {u'bash_template': u'/data2/avelt/Assembly_amurensis/4-quiver/quiver-split/bash-template.sh',
             u'units_of_work': u'/data2/avelt/Assembly_amurensis/4-quiver/quiver-chunks/000153Fp01_00001/some-units-of-work.json'},
 u'outputs': {u'results': u'results.json'},
 u'parameters': {u'pypeflow_mb': 4000, u'pypeflow_nproc': u'5'}}
2018-09-25 01:34:24,415 - root - WARNING - CD: '/data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001' <- '/data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001'
2018-09-25 01:34:24,415 - root - DEBUG - Checking existence of u'/data2/avelt/Assembly_amurensis/4-quiver/quiver-chunks/000153Fp01_00001/some-units-of-work.json' with timeout=30
2018-09-25 01:34:24,416 - root - DEBUG - Checking existence of u'/data2/avelt/Assembly_amurensis/4-quiver/quiver-split/bash-template.sh' with timeout=30
2018-09-25 01:34:24,416 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2018-09-25 01:34:24,416 - root - WARNING - CD: '/data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001' <- '/data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001'
2018-09-25 01:34:24,417 - root - INFO - $('/bin/bash user_script.sh')
hostname
+ hostname
pwd
+ pwd
date
+ date
# Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.generic_run_units_of_work --nproc=5 --units-of-work-fn=/data2/avelt/Assembly_amurensis/4-quiver/quiver-chunks/000153Fp01_00001/some-units-of-work.json --bash-template-fn=/data2/avel
t/Assembly_amurensis/4-quiver/quiver-split/bash-template.sh --results-fn=results.json
+ python -m falcon_kit.mains.generic_run_units_of_work --nproc=5 --units-of-work-fn=/data2/avelt/Assembly_amurensis/4-quiver/quiver-chunks/000153Fp01_00001/some-units-of-work.json --bash-template-fn=/data2/av
elt/Assembly_amurensis/4-quiver/quiver-split/bash-template.sh --results-fn=results.json
falcon-kit 1.2.2
pypeflow 2.0.4
INFO:root:INPUT:{u'ref_fasta': u'/data2/avelt/Assembly_amurensis/4-quiver/quiver-split/./refs/000153Fp01_00001/ref.fa', u'read_bam': u'/data2/avelt/Assembly_amurensis/4-quiver/segregate-run/segr1905/segregate
d/000153Fp01_00001/000153Fp01_00001.bam', u'ctg_type': u'/data2/avelt/Assembly_amurensis/4-quiver/quiver-split/./refs/000153Fp01_00001/ctg_type'}
INFO:root:OUTPUT:{u'cns_fasta': u'cns.fasta.gz', u'cns_vcf': u'cns.vcf', u'job_done': u'quiver_done', u'ctg_type_again': u'ctg_type', u'cns_fastq': u'cns.fastq.gz'}
INFO:root:PARAMS:{'pypeflow_nproc': '5', u'ctg_id': u'000153Fp01_00001'}
INFO:root:$('rm -rf uow-00')
WARNING:root:CD: 'uow-00' <- '/data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001'
INFO:root:$('/bin/bash user_script.sh')
hostname
+ hostname
pwd
+ pwd
date
+ date
set -vex
+ set -vex
trap 'touch quiver_done.exit' EXIT
+ trap 'touch quiver_done.exit' EXIT
hostname
+ hostname
date
+ date

samtools faidx /data2/avelt/Assembly_amurensis/4-quiver/quiver-split/./refs/000153Fp01_00001/ref.fa
+ samtools faidx /data2/avelt/Assembly_amurensis/4-quiver/quiver-split/./refs/000153Fp01_00001/ref.fa
[faidx] Could not build fai index /data2/avelt/Assembly_amurensis/4-quiver/quiver-split/./refs/000153Fp01_00001/ref.fa.fai
touch quiver_done.exit
+ touch quiver_done.exit
WARNING:root:Call '/bin/bash user_script.sh' returned 256.
WARNING:root:CD: 'uow-00' -> '/data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001'
Traceback (most recent call last):
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 115, in <module>
    main()
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 111, in main
    run(**vars(args))
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 64, in run
    pypeflow.do_task.run_bash(script, inputs, outputs, params)
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/pypeflow/do_task.py", line 178, in run_bash
    util.system(cmd)
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
    raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
2018-09-25 01:34:24,870 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2018-09-25 01:34:24,870 - root - WARNING - CD: '/data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001' -> '/data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001'
2018-09-25 01:34:24,870 - root - WARNING - CD: '/data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001' -> '/data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001'
2018-09-25 01:34:24,870 - root - CRITICAL - Error in /cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/data2/avelt/Assembly_amurensis/4-quiver/quiver-r
un/000153Fp01_00001/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/pypeflow/do_task.py", line 246, in <module>
    main()
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/pypeflow/do_task.py", line 238, in main
    run(**vars(parsed_args))
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/pypeflow/do_task.py", line 232, in run
    run_cfg_in_tmpdir(cfg, tmpdir)
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/pypeflow/do_task.py", line 208, in run_cfg_in_tmpdir
    run_bash(bash_template, myinputs, myoutputs, parameters)
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/pypeflow/do_task.py", line 178, in run_bash
    util.system(cmd)
  File "/cm/shared/apps/FALCON_12_09_2018/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
    raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
+++ pwd
++ echo 'FAILURE. Running top in /data2/avelt/Assembly_amurensis/4-quiver/quiver-run/000153Fp01_00001 (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get
diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
++ which python
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ pstree -apl

real    0m1.432s
user    0m0.327s
sys     0m0.164s
+ finish
+ echo 'finish code: 1'

To Reproduce
I give you the two fasta files at the end of the 3-unzip step, are they enough?
https://www.dropbox.com/s/l5n2gu2etdhkute/3-unzip.zip?dl=0

Expected behavior
I expect that two polished fasta files must be generate, one for primary contigs and one for haplotigs.

Best,
Amandine

Polishing with pb-assembly

Hi,

Running fc_unzip.py from the pb-assembly bioconda does unzipping + polishing and generates a folder named 4-quiver. Is it using the Quiver algorithm? Can I also make it use Arrow?

Also, is there a parameter list for the polishing step? The help for fc_unzip.py does not say a lot.

Thanks!
Gui

pb-falcon package does not depend on pypeflow package

Currently https://github.com/bioconda/bioconda-recipes/blob/master/recipes/pb-falcon/meta.yaml bundles https://github.com/PacificBiosciences/pypeFLOW/archive/2.1.0.tar.gz as part of the source code.

I think it should instead declare a dependency on https://github.com/bioconda/bioconda-recipes/blob/master/recipes/pypeflow/meta.yaml (which I note would first need updating to v2.1.0)

How to determine quality of a PacBio dataset?

In the documentation for Falcon and Falcon-Unzip, there are suggestions for the setting of the kmer and the e options for all the Dazzler programs where the choice depends upon the quality of the sequence data.

What's an objective way to determine high versus low quality in the context of setting these parameters?

Wrong homepage URL in pb-falcon package metadata

Wrong homepage URL in pb-falcon package metadata as shown on https://anaconda.org/bioconda/pb-falcon

Currently https://github.com/bioconda/bioconda-recipes/blob/master/recipes/pb-falcon/meta.yaml

about:
  home: 'https://github.com/PacificBiosciences/pypeFLOW'
  license: "BSD 3-Clause Clear License"
  summary: "FALCON/Unzip tool-suite (originally by Jason Chin)"
  authors:
  - cschin
  - pb-cdunn
  - isovic

I'm unclear what the best URL would be, maybe https://github.com/PacificBiosciences/FALCON or https://pb-falcon.readthedocs.io/ ?

Updating metadata for pbbam package

Re: https://github.com/bioconda/bioconda-recipes/blob/master/recipes/pbbam/meta.yaml

about:
  home: https://github.com/PacificBiosciences/pbbam
  license: BSD
  summary: PacBio BAM C++ library

When installed this says BSD 3-Clause Clear License rather than just BSD, consistent with https://github.com/PacificBiosciences/pbbam/blob/develop/LICENSE.txt

P.S. Also it would be nice to have a more details description as the about summary, e.g. Copied from https://github.com/PacificBiosciences/pbbam/blob/develop/README.md says:

The pbbam software package provides components to create, query, & edit PacBio BAM files and associated indices. These components include a core C++ library, bindings for additional languages, and command-line utilities.

e.g. "C++ library and command line tools to create, query, edit and index PacBio BAM files."

REPmask error at very beginning step

I'm not sure if this is a bug or my carelessness.

My job terminated at the very beginning step and returns

[ERROR]Task Node(0-rawreads/repa/rep-runs/rep_053) failed with exit-code=1
[ERROR]Task Node(0-rawreads/repa/rep-runs/rep_040) failed with exit-code=1
[ERROR]Some tasks are recently_done but not satisfied: set([Node(0-rawreads/repa/rep-runs/rep_053), Node(0-rawreads/repa/rep-runs/rep_040)])
[ERROR]ready: set([Node(0-rawreads/repa/rep-runs/rep_019), Node(0-rawreads/repa/rep-runs/rep_061), Node(0-rawreads/repa/rep-runs/rep_038), Node(0-rawreads/repa/rep-runs/rep_009), Node(0-rawreads/repa/rep-runs/rep_049), Node(0-rawreads/repa/rep-chunks/rep_030), Node(0-rawreads/repa/rep-chunks/rep_059), Node(0-rawreads/repa/rep-chunks/rep_031), Node(0-rawreads/repa/rep-chunks/rep_048), Node(0-rawreads/repa/rep-chunks/rep_055), Node(0-rawreads/repa/rep-chunks/rep_023), Node(0-rawreads/repa/rep-runs/rep_032), Node(0-rawreads/repa/rep-chunks/rep_037), Node(0-rawreads/repa/rep-chunks/rep_014), Node(0-rawreads/repa/rep-chunks/rep_027), Node(0-rawreads/repa/rep-chunks/rep_010), Node(0-rawreads/repa/rep-chunks/rep_070), Node(0-rawreads/repa/rep-runs/rep_057), Node(0-rawreads/repa/rep-chunks/rep_007), Node(0-rawreads/repa/rep-chunks/rep_035), Node(0-rawreads/repa/rep-chunks/rep_033), Node(0-rawreads/repa/rep-runs/rep_054), Node(0-rawreads/repa/rep-chunks/rep_045), Node(0-rawreads/repa/rep-chunks/rep_052), Node(0-rawreads/repa/rep-chunks/rep_066), Node(0-rawreads/repa/rep-chunks/rep_024), Node(0-rawreads/repa/rep-chunks/rep_050), Node(0-rawreads/repa/rep-chunks/rep_006), Node(0-rawreads/repa/rep-chunks/rep_069), Node(0-rawreads/repa/rep-chunks/rep_011), Node(0-rawreads/repa/rep-chunks/rep_022), Node(0-rawreads/repa/rep-chunks/rep_043), Node(0-rawreads/repa/rep-chunks/rep_004), Node(0-rawreads/repa/rep-chunks/rep_068), Node(0-rawreads/repa/rep-chunks/rep_047), Node(0-rawreads/repa/rep-chunks/rep_026), Node(0-rawreads/repa/rep-chunks/rep_056), Node(0-rawreads/repa/rep-chunks/rep_001), Node(0-rawreads/repa/rep-chunks/rep_046), Node(0-rawreads/repa/rep-chunks/rep_002), Node(0-rawreads/repa/rep-chunks/rep_018), Node(0-rawreads/repa/rep-chunks/rep_034), Node(0-rawreads/repa/rep-chunks/rep_025), Node(0-rawreads/repa/rep-chunks/rep_039), Node(0-rawreads/repa/rep-chunks/rep_064), Node(0-rawreads/repa/rep-runs/rep_017), Node(0-rawreads/repa/rep-chunks/rep_065), Node(0-rawreads/repa/rep-chunks/rep_063), Node(0-rawreads/repa/rep-chunks/rep_013), Node(0-rawreads/repa/rep-runs/rep_041), Node(0-rawreads/repa/rep-chunks/rep_067), Node(0-rawreads/repa/rep-chunks/rep_012), Node(0-rawreads/repa/rep-chunks/rep_021), Node(0-rawreads/repa/rep-runs/rep_016), Node(0-rawreads/repa/rep-chunks/rep_060), Node(0-rawreads/repa/rep-chunks/rep_015), Node(0-rawreads/repa/rep-chunks/rep_020), Node(0-rawreads/repa/rep-chunks/rep_062), Node(0-rawreads/repa/rep-chunks/rep_071), Node(0-rawreads/repa/rep-chunks/rep_028), Node(0-rawreads/repa/rep-chunks/rep_000), Node(0-rawreads/repa/rep-chunks/rep_058), Node(0-rawreads/repa/rep-chunks/rep_044), Node(0-rawreads/repa/rep-chunks/rep_005)])
        submitted: set([Node(0-rawreads/repa/rep-chunks/rep_042), Node(0-rawreads/repa/rep-chunks/rep_008), Node(0-rawreads/repa/rep-chunks/rep_036), Node(0-rawreads/repa/rep-chunks/rep_051), Node(0-rawreads/repa/rep-chunks/rep_029), Node(0-rawreads/repa/rep-runs/rep_003)])
[ERROR]Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.

Not every chunk returns error.

I look into file 0-rawreads/repa/rep-runs/rep_053/run-P4858d5ae2546fb.bash.stderr, found

[INFO]$('bash -vex run_REPmask.sh')
REPmask -v -c0 -nrep0 raw_reads xxxxxx/falcon/run2/0-rawreads/repa/las-merge-gathered/../las-merge-runs/m_00053/uow-00/raw_reads.54.las
+ REPmask -v -c0 -nrep0 raw_reads xxxxxx/falcon/run2/0-rawreads/repa/las-merge-gathered/../las-merge-runs/m_00053/uow-00/raw_reads.54.las
REPmask: Repeat coverage threshold must be positive (0)
[WARNING]Call 'bash -vex run_REPmask.sh' returned 256.

look like REPmask error?

pb-assembly v0.0.1 is installed via miniconda on a ubuntu machine.

isoseq3 package not found at bioconda

Operating system
MacOS Sierra 10.13.4

Package name
The Anaconda3 was installed using graphical installer
Conda packages were updated
All the installation steps available at https://github.com/PacificBiosciences/IsoSeq_SA3nUP/wiki/Tutorial:-Installing-and-Running-Iso-Seq-3-using-Conda were successfully done, but the installation of isoseq3.

Describe the bug
isoseq3 package was not found at bioconda channel

Error message
(anaCogent5.2) MBP-de-Daniele:~ dani$ conda install -n anaCogent5.2 -c bioconda isoseq3
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

isoseq3

Current channels:

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page

Expected behavior
All the optional packages below were successfully installed:
conda install -n anaCogent5.2 -c bioconda pbcoretools # for manipulating PacBio datasets
conda install -n anaCogent5.2 -c bioconda bamtools # for converting BAM to fasta
conda install -n anaCogent5.2 -c bioconda pysam # for making CSV reports

Thank you in advance for any help,
Daniele

Config File for falcon_phase.py with HiC data

Hello,

We are interested in using Falcon-PHASE with HiC data for phasing. Is there an example config file available for the falcon_phase.py script? Thank you very much!

error occurred in 3-unzip/2-htigs step of falcon_unzip.

Error message
the message in 3-unzip/2-htigs/chunk_002716F/run-P3105148f148918.bash.stderr:
[ERROR 2018-11-05 02:39:40] Failure in generate_haplotigs_for_ctg((u'002716F', u'../../0-phasing/002716F/uow-00/proto', './uow-002716F', u'../../..', False)) Traceback (most recent call last): File "~/miniconda2/envs/assembly/lib/python2.7/site-packages/falcon_unzip/mains/graphs_to_h_tigs_2.py", line 74, in run_generate_haplotigs_for_ctg

.........skipping...........

raise Exception(msg) Exception: Skipping additional subgraphs of the primary contig: 002716F. The graph has multiple primary components.

How can I fix it? Or, how to skip the contig(002716F)? Actually, I have three similar contig that make me can't go on .

bax2bam: symbol lookup error: ../lib/libblasr.so.5.3.1: undefined symbol: _ZNK2H510H5Location15getObjnameByIdxEy

Operating system
4.4.0-119-generic #143-Ubuntu with anaconda 1.6.14

Package name
bax2bam

Describe the bug
bax2bam throws a lookup error.

Error message
bax2bam: symbol lookup error: ../lib/libblasr.so.5.3.1: undefined symbol: _ZNK2H510H5Location15getObjnameByIdxEy

To Reproduce
conda install -c bioconda
bam2bax -f fofn

Expected behavior
bax2bam should convert my bax to bam ;)

blasr bam output problem

I tried to do alignment using blasr. Blasr seems to run fine and gave no error message yet the output bam file is unreable like this:

unix306 631 % blasr ~/data/lambda_phage/lp1d.fasta  ~/data/lambda_phage/lambda_phage_ref.fasta --bam --out aligned.bam
[INFO] 2018-09-24T12:12:18 [blasr] started.
[INFO] 2018-09-24T12:12:19 [blasr] ended.
unix306 632 % cat aligned.bam
?BC\?QKn?@
          ?ˉz?\ ????FBBH?U?    A~]?n?HOSR?U??�??g[~???|?H???]?(Z?8x
V?:s?b???Q?`?W??&?
                  ?vG?)?)??=O?Jg?NaJ]׉ʁ44P???<???@?
                                                   4z@??Bka/Wȉ?Ƶ??K?ΣؾĆ#
??j??'?O??l?tƲjEsE???E??!,k#?y??]??1???R???ie(???4?ߎPe8??@.??        ??GȄ\?I?S?P??]Ǯ72ï? 
?}k'??K?'?@???BCnix306 633 % Sߍ?ȳJ$?R

bax2bam: libpbbam.so.0.18.0 is missing

Running bax2bam fails with the following error:
bax2bam: error while loading shared libraries: libpbbam.so.0.18.0: cannot open shared object file: No such file or directory.

To reproduce:

OS: Debian

Environment.yaml

channels:
  - bioconda
dependencies:
  - python=2.7
  - bax2bam=0.0.9

Create environment:

conda env create -n test -f environment.yaml

Activate environment

source activate test

Run bax2bam

bax2bam --version
>> bax2bam: error while loading shared libraries: libpbbam.so.0.18.0: cannot open shared object file: No such file or directory

Further info:

Looking in the environment/lib directory, the following are present:
libpbbam.so which is a symlink to libpbbam.so.0.19.0

So it appears that the supplied bax2bam binary was compiled against an older version of libpbbam than the version installed.

Even further info:

Downgrading the pbbam package to 0.18.0 after installation of bax2bam allows bax2bam to be used

conda install -c bioconda pbbam=0.18.0
bax2bam --version
>>0.0.8

(note that bax2bam now reports being version 0.0.8 after downgrading pbbam)

`pa_DBdust_option` will not pass the option to DBdust

Operating system

CentOS Linux release 7.5.1804 (Core)

Package name

pb-assembly

Conda environment

my conda environment is

# packages in environment at /opt/biosoft/pb-assembly:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py27_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
avro-python2              1.8.2                      py_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
bcftools                  1.9                  h4da6232_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
bedtools                  2.27.1               he941832_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
blas                      1.0                         mkl    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
blasr                     5.3.2                hac9d22c_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
blasr_libcpp              5.3.1                hac9d22c_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
bwa                       0.7.17               ha92aebf_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
bzip2                     1.0.6                h470a237_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
ca-certificates           2018.8.24            ha4d7672_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
certifi                   2018.8.24             py27_1001    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
cffi                      1.11.5           py27h5e8e0c9_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
chardet                   3.0.4                    py27_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
cryptography              2.3.1            py27hdffb7b8_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
cryptography-vectors      2.3.1                    py27_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
curl                      7.61.0               h93b3f91_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
cython                    0.28.5           py27hfc679d8_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
decorator                 4.3.0                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
enum34                    1.1.6                    py27_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
future                    0.16.0                   py27_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
genomicconsensus          2.3.2                    py27_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
h5py                      2.8.0            py27hb794570_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
hdf5                      1.10.2               hc401514_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
htslib                    1.7                           0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
idna                      2.7                      py27_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
intel-openmp              2019.0                      118    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ipaddress                 1.0.22                     py_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
iso8601                   0.1.12                     py_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
krb5                      1.14.6                        0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libdeflate                1.0                  h470a237_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
libffi                    3.2.1                hfc679d8_5    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libgcc                    7.2.0                h69d50b8_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libgcc-ng                 7.2.0                hdf63c60_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libgfortran               3.0.0                         1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libssh2                   1.8.0                h5b517e9_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
linecache2                1.0.0                    py27_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
minimap2                  2.12                 ha92aebf_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
mkl                       2019.0                      118    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
mkl_fft                   1.0.6                    py27_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
mkl_random                1.0.1                    py27_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
mummer4                   4.0.0beta2      pl526hfc679d8_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
ncurses                   6.1                  hfc679d8_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
networkx                  2.2                        py_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
nim-falcon                0.0.0                         0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
numpy                     1.15.0           py27h1b885b7_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
numpy-base                1.15.0           py27h3dfced4_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
openssl                   1.0.2p               h470a237_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
pb-assembly               0.0.1                    py27_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
pb-dazzler                0.0.0                h470a237_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
pb-falcon                 0.2.3            py27ha92aebf_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
pbalign                   0.3.1                    py27_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
pbbam                     0.18.0               h1310cd9_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
pbcommand                 1.1.1                    py27_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
pbcore                    1.5.1                    py27_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
perl                      5.26.2               h470a237_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
pip                       18.0                     py27_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
pycparser                 2.19                       py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
pyopenssl                 18.0.0                   py27_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
pysam                     0.14.1           py27hae42fb6_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
pysocks                   1.6.8                    py27_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
python                    2.7.15               h9fef7bc_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
python-consensuscore      1.1.1            py27h02d93b8_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
python-consensuscore2     3.1.0                    py27_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
python-edlib              1.2.3            py27h470a237_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
python-intervaltree       2.1.0                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
python-msgpack            0.5.6            py27h470a237_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
python-sortedcontainers   2.0.4                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
pytz                      2018.5                     py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
readline                  7.0                  haf1bffa_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
requests                  2.19.1                   py27_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
samtools                  1.9                  h8ee4bcc_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda
setuptools                40.4.0                py27_1000    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
six                       1.11.0                   py27_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
sqlite                    3.25.1               hb1c47c0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
tk                        8.6.8                ha92aebf_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
traceback2                1.4.0                    py27_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
unittest2                 1.1.0                      py_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
urllib3                   1.23                     py27_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
wheel                     0.31.1                py27_1001    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
xz                        5.2.4                h470a237_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
zlib                      1.2.11               h470a237_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge

Describe the bug

pa_DBdust_option will not use the options I set. I use the E.coli data from https://pb-falcon.readthedocs.io/en/latest/tutorial.html , and I set this option to pa_DBdust_option=-w66 -m12, and keep other options same as https://github.com/PacificBiosciences/pb-assembly/blob/master/cfgs/fc_run_200kb.cfg.

And this is General_config.json

cat General_config.json 
{
  "ovlp_HPCTANmask_option": "-l500",
  "falcon_sense_greedy": false,
  "pa_fasta_filter_option": "pass",
  "seed_coverage": "20",
  "overlap_filtering_setting": "--max-diff 100 --max-cov 100 --min-cov 2",
  "length_cutoff_pr": "1000",
  "input_fofn": "input.fofn",
  "pa_HPCdaligner_option": "-v -B128 -M24",
  "ovlp_daligner_option": "-e.9 -l2500 -k24 -h1024 -w6 -s100",
  "bestn": 12,
  "pa_HPCTANmask_option": "",
  "input_type": "raw",
  "pa_REPmask_code": "0,300;0,300;0,300",
  "pa_DBsplit_option": "-x500 -s200",
  "fc_ovlp_to_graph_option": " --min-len 1000",
  "genome_size": "0",
  "avoid_text_file_busy": true,
  "pa_dazcon_option": "-j 4 -x -l 500",
  "skip_checks": false,
  "ver": "100",
  "target": "assembly",
  "falcon_sense_skip_contained": false,
  "pa_daligner_option": "-e.8 -l2000 -k18 -h480  -w8 -s100",
  "length_cutoff": "3000",
  "pa_DBdust_option": "-w66 -m12",
  "ovlp_HPCdaligner_option": "-v -B128 -M24",
  "LA4Falcon_preload": false,
  "dazcon": false,
  "falcon_sense_option": "--output-multi --min-idt 0.70 --min-cov 2 --max-n-read 1800",
  "ovlp_DBsplit_option": "-x500 -s200"
}

and I use ps aux | grep DBdust to check whether the options is passed to DBdust.

Expected behavior

I think the behavior should be same as daligner, which I use ps aux | grep daligner get the following information. Which is in line with the options pa_HPCdaligner_option=-v -B128 -M24 and
pa_daligner_option=-e.8 -l2000 -k18 -h480 -w8 -s100

daligner -v -k18 -w8 -h480 -H3000 -e0.8 -l2000 -M24 -P. -mdust -mtan -mrep0 raw_reads.5 raw_reads.1 raw_reads.2 raw_reads.3 raw_reads.4 raw_reads.5

Phase 0 repmask code not using the dust track

I am attempting to assemble a large genome using the latest pb-assembly, and I noticed the the daligner commands for the initial repmask step do not reference the dust mask created earlier. An example daligner command line is

daligner -v -w8 -h480 -e0.75 -l3200 -M40 -P. raw_reads.85 raw_reads.81 raw_reads.82

and it should include -mdust.

falcon unzip example config needs update

Operating system
Linux specifically RHEL 7.4

Package name
FALCON-unzip

Describe the bug
The current version of FALCON unzip uses minimap2 rather than blasr in the phasing step, significantly reducing resource requirements and allowing greater parallelism. The [job.step.unzip.blasr_aln] section of https://github.com/PacificBiosciences/pb-assembly/blob/master/cfgs/fc_unzip.cfg does not reflect this and constrains the pipeline to running 2 jobs at a time whilst requesting large amounts of memory.

Expected behavior
The FALCON unzip example config should reflect resource requirements typical of the current version of the pipeline.

Bug in restarting falcon_unzip/unzip.pl

When restarting fc_unzip.py at an early stage, I get the following error traceback:

[INFO] ln -s 4-polish 4-quiver
Traceback (most recent call last):
  File "/db/congenomics/local6/binaries/conda/bin/fc_unzip.py", line 11, in <module>
    load_entry_point('falcon-unzip==1.1.3', 'console_scripts', 'fc_unzip.py')()
  File "/db/congenomics/local6/binaries/conda/lib/python2.7/site-packages/falcon_unzip/mains/start_unzip.py", line 29, in main
    unzip.run(**vars(args))
  File "/db/congenomics/local6/binaries/conda/lib/python2.7/site-packages/falcon_unzip/unzip.py", line 145, in run
    backward_compatible_dirs()
  File "/db/congenomics/local6/binaries/conda/lib/python2.7/site-packages/falcon_unzip/unzip.py", line 137, in backward_compatible_dirs
    symlink_if_missing(polish_dn, quiver_dn)
  File "/db/congenomics/local6/binaries/conda/lib/python2.7/site-packages/falcon_unzip/unzip.py", line 107, in symlink_if_missing
    os.symlink(src, name)
OSError: [Errno 17] File exists

The problem is that the "4-polish" folder doesn't exist yet. Thus, in the symlink_if_missing function, the test for 4-quiver fails, and then the code tries to recreate the symbolic link, and that already exists.

The solution is to change the test from "exists" to "lexists". Here's the updated code snippet:

def symlink_if_missing(src, name):
    if not os.path.lexists(name):
        LOG.info(' ln -s {} {}'.format(src, name))
        os.symlink(src, name)

Issue with FASTA reference for arrow

Hi,

I'm running iterations of Arrow and Pilon, and I'm running into some sort of problem with my FASTA or .fai reference from the output of Pilon (which I have modified to restore some large INDELs corrected by Pilon). I've done simple checks to validate the FASTA file and all looks ok. Any help would be much appreciated!

Command is:

arrow -j16 c_incerta_canu_v2.pbalign_v2.bam -r ../pilon_1/c_incerta.canu_v2.arrow_pilon_v1.fa -o c_incerta.canu_v2.arrow_v2.fa -o c_incerta.canu_v2.arrow_v2.fq -o c_incerta.canu_v2.arrow_v2.variants.gff

Error message is:

  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcommand/cli/core.py", line 138, in _pacbio_main_runner
    return_code = exe_main_func(*args, **kwargs)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/GenomicConsensus/main.py", line 340, in args_runner
    return tr.main()
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/GenomicConsensus/main.py", line 267, in main
    self._loadReference(peekFile)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/GenomicConsensus/main.py", line 125, in _loadReference
    reference.loadFromFile(options.referenceFilename, alnFile)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/GenomicConsensus/reference.py", line 97, in loadFromFile
    f = ReferenceSet(filename_)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 4336, in __init__
    super(ReferenceSet, self).__init__(*files, **kwargs)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 3887, in __init__
    super(ContigSet, self).__init__(*files, **kwargs)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 450, in __init__
    self.updateCounts()
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 4092, in updateCounts
    if not self.isIndexed:
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 4241, in isIndexed
    lambda x: isinstance(x, IndexedFastaReader))
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 1832, in _pollResources
    return [func(resource) for resource in self.resourceReaders()]
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 4200, in resourceReaders
    self._openFiles()
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 4155, in _openFiles
    resource = self._openFile(urlparse(extRes.resourceId).path)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 4170, in _openFile
    resource = IndexedFastaReader(location)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/FastaIO.py", line 410, in __init__
    self.fai = loadFastaIndex(self.faiFilename, self.view)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/FastaIO.py", line 267, in loadFastaIndex
    assert (header_[0] == ">" and header_[-1] == "\n")
AssertionError
[ERROR]
Traceback (most recent call last):
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcommand/cli/core.py", line 138, in _pacbio_main_runner
    return_code = exe_main_func(*args, **kwargs)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/GenomicConsensus/main.py", line 340, in args_runner
    return tr.main()
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/GenomicConsensus/main.py", line 267, in main
    self._loadReference(peekFile)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/GenomicConsensus/main.py", line 125, in _loadReference
    reference.loadFromFile(options.referenceFilename, alnFile)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/GenomicConsensus/reference.py", line 97, in loadFromFile
    f = ReferenceSet(filename_)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 4336, in __init__
    super(ReferenceSet, self).__init__(*files, **kwargs)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 3887, in __init__
    super(ContigSet, self).__init__(*files, **kwargs)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 450, in __init__
    self.updateCounts()
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 4092, in updateCounts
    if not self.isIndexed:
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 4241, in isIndexed
    lambda x: isinstance(x, IndexedFastaReader))
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 1832, in _pollResources
    return [func(resource) for resource in self.resourceReaders()]
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 4200, in resourceReaders
    self._openFiles()
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 4155, in _openFiles
    resource = self._openFile(urlparse(extRes.resourceId).path)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/dataset/DataSetIO.py", line 4170, in _openFile
    resource = IndexedFastaReader(location)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/FastaIO.py", line 410, in __init__
    self.fai = loadFastaIndex(self.faiFilename, self.view)
  File "/home/craigror/anaconda2/lib/python2.7/site-packages/pbcore/io/FastaIO.py", line 267, in loadFastaIndex
    assert (header_[0] == ">" and header_[-1] == "\n")
AssertionError```

Missing newlines in 0-rawreads/daligner-split/run_jobs.sh

In a test run of pb-assembly, I found a shell script that looks like it's missing newlines between each command. The file was in

0-rawreads/daligner-split

and named run_jobs.sh

and the only line begins like this:

rm raw_reads.1.raw_reads.*.lasrm raw_reads.2.raw_reads.*.lasrm raw_reads.3.raw_reads.*.lasrm

pb-assembly falcon does not run with slurm

Operating system
CentOS Linux release 7.3.1611

Package name
pb-assembly installed today via anaconda

Describe the bug
I am trying to configure falcon for slurm, but keep getting the error message: TypeError: object.__init__() takes no parameters - see below

Error message

[INFO] starting job Job(jobid='Pe1c6df12cc584e', cmd='/bin/bash run.sh', rundir='/home/lehmanr/melpacbio/0-assembly/newFalcon/0-rawreads/build', options={'JOB_QUEUE': 'batch', 'pwatcher_type': '', 'use_tmpdir': False, 'MB': 4000, 'job_type': 'slurm', 'submit': 'sbatch  \\\n-p ${JOB_QUEUE}     \\\n--job-name=${JOB_NAME}      \\\n-o "${JOB_STDOUT}"  \\\n-e "${JOB_STDERR}"  \\\n--cpus-per-task ${NPROC}    \\\n--mem ${MB}    \\\n--time ${time} \\\n--nodes 1 --ntasks 1 \\\n"${JOB_SCRIPT}"', 'NPROC': 1, 'njobs': '32'}) w/ job_type=SLURM
Traceback (most recent call last):
  File "/home/lehmanr/.conda/envs/denovo_asm/bin/fc_run.py", line 11, in <module>
    load_entry_point('falcon-kit==1.2.2', 'console_scripts', 'fc_run.py')()
  File "/home/lehmanr/.conda/envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 724, in main
    main1(argv[0], args.config, args.logger)
  File "/home/lehmanr/.conda/envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 76, in main1
    input_fofn_fn=input_fofn_fn,
  File "/home/lehmanr/.conda/envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 242, in run
    dist=Dist(NPROC=4, MB=4000, job_dict=config['job.step.da']),
  File "/home/lehmanr/.conda/envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/pype.py", line 106, in gen_parallel_tasks
    wf.refreshTargets()
  File "/home/lehmanr/.conda/envs/denovo_asm/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 277, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/home/lehmanr/.conda/envs/denovo_asm/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 316, in _refreshTargets
    unsubmitted = set(self.tq.enque(to_submit)) # In theory, this traps exceptions.
  File "/home/lehmanr/.conda/envs/denovo_asm/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 152, in enque
    result = self.watcher.run(**watcher_args)
  File "/home/lehmanr/.conda/envs/denovo_asm/lib/python2.7/site-packages/pwatcher/fs_based.py", line 650, in run
    return cmd_run(self.state, jobids, job_type, job_defaults_dict)
  File "/home/lehmanr/.conda/envs/denovo_asm/lib/python2.7/site-packages/pwatcher/fs_based.py", line 453, in cmd_run
    bjob = MetaJobSlurm(mjob)
  File "/home/lehmanr/.conda/envs/denovo_asm/lib/python2.7/site-packages/pwatcher/fs_based.py", line 398, in __init__
    super(MetaJobSlurm, self).__init__(mjob)
TypeError: object.__init__() takes no parameters

To Reproduce
My relevant fc_run.cfg:

[job.defaults]
job_type=slurm
pwatcher_type=
JOB_QUEUE=batch
MB=32768
NPROC=6
njobs=32
time=08:00:00
submit = sbatch  \
  -p ${JOB_QUEUE}     \
  --job-name=${JOB_NAME}      \
  -o "${JOB_STDOUT}"  \
  -e "${JOB_STDERR}"  \
  --cpus-per-task ${NPROC}    \
  --mem ${MB}    \
  --time ${time} \
  --nodes 1 --ntasks 1 \
  "${JOB_SCRIPT}"

PBdust should be parallelized

The initial call to PBdust after building the Dazzler database should be parallelized.

I'm currently working on a large genome, and the initial DBdust step takes about 10 hours because it's a single, serial task. It could be parallelized across the database blocks and the Catrack'ed together. From my own experience, this can work in minutes assuming that Falcon can run a few hundred jobs at once.

FALCON resource allocation

Hello,

I have read several threads on this but I am still struggling with resource allocation on the FALCON config file for PB-assembly.

First, how do the number of jobs and processors in the [job.defaults] section correlates with the --n_core parameter in the falcon_sense_option and overlap_filtering_setting?
Second, what happens if I do not set a limit memory per processor using the MB = parameter?

In general I want to maximize resource usage in local mode to assemble a fly genome (180 Mb). I have a 48 core machine with ~500Gb of RAM.
Bellow is my latest config file, any help is welcomed.

[General]
# list of files of the initial bas.h5 files
input_fofn = input.fofn

input_type = raw

# The length cutoff used for seed reads used for initial mapping
length_cutoff = -1
genome_size = 14200000
seed_coverage = 30

# The length cutoff used for seed reads usef for pre-assembly
length_cutoff_pr = 1000

pa_daligner_option   = -e.70 -l2000 -k18 -h240 -w8 -s100
ovlp_daligner_option = -e.96 -l1000 -k24 -h240 -w6 -s100
pa_HPCdaligner_option   = -v -B128 -M48
ovlp_HPCdaligner_option = -v -B128 -M48

pa_DBsplit_option = -x500 -s400
ovlp_DBsplit_option = -s400

falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 2 --max_n_read 200 --n_core 12
falcon_sense_skip_contained = True

overlap_filtering_setting = --max-diff 100 --max-cov 100 --min-cov 2 --n-core 24

[job.defaults]
job_type = local

use_tmpdir = /lscratch
pwatcher_type = blocking
job_type = string
submit = bash -C ${CMD} >| ${STDOUT_FILE} 2>| ${STDERR_FILE}

NPROC = 48
njobs = 1
MB = 50000
[job.step.da]
[job.step.pda]
[job.step.la]
[job.step.pla]
[job.step.cns]
[job.step.asm]

Arrow warning: Some reference contigs aligned against are not found in the reference FASTA.

Operating system
Which operating system and version are you using?

$ cat /etc/centos-release
CentOS Linux release 7.5.1804 (Core) 
$ arch
x86_64

Package name
Which package / tool is causing the problem? Which version are you using, use tool --version. Have you updated to the latest version conda update package?

Using arrow from GenomicConsensus,

$ arrow --version
2.3.2

Conda environment
What is the result of conda list? (Try to paste that between triple backticks.)

$ conda list
# packages in environment at /mnt/.../apps/conda:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                py27_1003    conda-forge
avro-python2              1.8.2                      py_1    bioconda
bcftools                  1.9                  h4da6232_0    bioconda
biopython                 1.68                     py27_0    bioconda
blas                      1.0                         mkl  
blast                     2.7.1                h4422958_6    bioconda
boost                     1.67.0           py27h3e44d54_0    conda-forge
boost-cpp                 1.67.0               h3a22d5f_0    conda-forge
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.10.15           ha4d7672_0    conda-forge
cairo                     1.14.12              he6fea26_5    conda-forge
certifi                   2018.10.15            py27_1000    conda-forge
cffi                      1.11.5           py27h5e8e0c9_1    conda-forge
chardet                   3.0.4                 py27_1003    conda-forge
conda                     4.5.11                py27_1000    conda-forge
conda-env                 2.6.0                         1  
cryptography              2.3.1            py27hdffb7b8_0    conda-forge
cryptography-vectors      2.3.1                 py27_1000    conda-forge
curl                      7.61.1               h74213dd_2    conda-forge
cython                    0.29             py27hfc679d8_0    conda-forge
emacs                     26.1                 h2e30b44_2    conda-forge
enum34                    1.1.6                 py27_1001    conda-forge
fontconfig                2.13.1               h65d0f4c_0    conda-forge
freetype                  2.9.1                h6debe1e_4    conda-forge
futures                   3.2.0                 py27_1000    conda-forge
genomicconsensus          2.3.2                    py27_3    bioconda
gettext                   0.19.8.1             h5e8e0c9_1    conda-forge
giflib                    5.1.4                h470a237_1    conda-forge
glib                      2.55.0               h464dc38_2    conda-forge
gmp                       6.1.2                hfc679d8_0    conda-forge
gnutls                    3.5.19               h2a4e5f8_1    conda-forge
h5py                      2.8.0            py27h097b052_4    conda-forge
hdf5                      1.10.3               hc401514_2    conda-forge
htslib                    1.9                  hc238db4_4    bioconda
icu                       58.2                 hfc679d8_0    conda-forge
idna                      2.7                   py27_1002    conda-forge
intel-openmp              2019.0                      118  
ipaddress                 1.0.22                     py_1    conda-forge
iso8601                   0.1.12                     py_1    conda-forge
jpeg                      9c                   h470a237_1    conda-forge
krb5                      1.16.1               hbb41f41_0    conda-forge
libcurl                   7.61.1               hbdb9355_2    conda-forge
libdeflate                1.0                  h470a237_0    bioconda
libedit                   3.1.20170329         h6b74fdf_2  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libiconv                  1.15                 h470a237_3    conda-forge
libpng                    1.6.35               ha92aebf_2    conda-forge
libssh2                   1.8.0                h5b517e9_2    conda-forge
libstdcxx-ng              8.2.0                hdf63c60_1  
libtiff                   4.0.9                he6b73bb_2    conda-forge
libuuid                   2.32.1               h470a237_2    conda-forge
libxcb                    1.13                 h470a237_2    conda-forge
libxml2                   2.9.8                h422b904_5    conda-forge
linecache2                1.0.0                      py_1    conda-forge
mkl                       2019.0                      118  
mkl_fft                   1.0.6                    py27_0    conda-forge
mkl_random                1.0.1                    py27_0    conda-forge
mmtf-python               1.1.2                      py_0    conda-forge
msgpack-python            0.5.6            py27h2d50403_3    conda-forge
ncurses                   6.1                  hfc679d8_1    conda-forge
nettle                    3.3                           0    conda-forge
numpy                     1.15.3           py27h1d66e8a_0  
numpy-base                1.15.3           py27h81de0dd_0  
olefile                   0.46                       py_0    conda-forge
openjpeg                  2.3.0                h0e734dc_3    conda-forge
openssl                   1.0.2p               h470a237_1    conda-forge
pbbam                     0.19.0               h6678c95_0    bioconda
pbcommand                 1.1.1                    py27_2    bioconda
pbcore                    1.6.5                    py27_0    bioconda
pcre                      8.41                 hfc679d8_3    conda-forge
perl                      5.26.2               h470a237_0    conda-forge
perl-archive-tar          2.18                    pl526_3    bioconda
perl-carp                 1.38                    pl526_1    bioconda
perl-compress-raw-bzip2   2.074           pl526hfc679d8_0    bioconda
perl-compress-raw-zlib    2.081           pl526h2d50403_0    bioconda
perl-data-dumper          2.161                   pl526_2    bioconda
perl-encode               2.88                    pl526_1    bioconda
perl-exporter             5.72                    pl526_1    bioconda
perl-exporter-tiny        1.000000                pl526_0    bioconda
perl-extutils-makemaker   7.34                    pl526_2    bioconda
perl-io-compress          2.069           pl526hfc679d8_5    bioconda
perl-io-zlib              1.10                    pl526_2    bioconda
perl-list-moreutils       0.428                   pl526_1    bioconda
perl-list-moreutils-xs    0.428                   pl526_0    bioconda
perl-parent               0.236                   pl526_1    bioconda
perl-pathtools            3.73                 h470a237_2    bioconda
perl-scalar-list-utils    1.45            pl526h470a237_3    bioconda
perl-test-more            1.001002                pl526_1    bioconda
perl-xsloader             0.24                    pl526_0    bioconda
pillow                    5.3.0            py27hc736899_0    conda-forge
pip                       18.1                  py27_1000    conda-forge
pixman                    0.34.0               h470a237_3    conda-forge
poppler                   0.67.0               h4d7e492_3    conda-forge
poppler-data              0.4.9                         0    conda-forge
pthread-stubs             0.4                  h470a237_1    conda-forge
pycosat                   0.6.3            py27h470a237_1    conda-forge
pycparser                 2.19                       py_0    conda-forge
pyopenssl                 18.0.0                py27_1000    conda-forge
pysam                     0.15.1           py27h0380709_0    bioconda
pysocks                   1.6.8                 py27_1002    conda-forge
python                    2.7.15               h9fef7bc_0    conda-forge
python-consensuscore      1.1.1            py27h02d93b8_2    bioconda
python-consensuscore2     3.1.0                    py27_2    bioconda
pytz                      2018.6                     py_0    conda-forge
readline                  7.0                  h7b6447c_5  
reportlab                 3.5.9            py27h77bcf2c_0    conda-forge
requests                  2.13.0                   py27_0    conda-forge
ruamel_yaml               0.15.71          py27h470a237_0    conda-forge
samtools                  1.9                  h8ee4bcc_1    bioconda
setuptools                40.5.0                   py27_0    conda-forge
six                       1.11.0                py27_1001    conda-forge
sqlite                    3.24.0               h84994c4_0  
tk                        8.6.8                hbc83047_0  
traceback2                1.4.0                    py27_0    conda-forge
unittest2                 1.1.0                      py_0    conda-forge
urllib3                   1.12                     py27_0    bioconda
wheel                     0.32.2                   py27_0    conda-forge
xorg-fixesproto           5.0                  h470a237_2    conda-forge
xorg-kbproto              1.0.7                h470a237_2    conda-forge
xorg-libice               1.0.9                h470a237_4    conda-forge
xorg-libsm                1.2.3                h8c8a85c_0    conda-forge
xorg-libx11               1.6.6                h470a237_0    conda-forge
xorg-libxau               1.0.8                h470a237_6    conda-forge
xorg-libxaw               1.0.13               h470a237_2    conda-forge
xorg-libxdmcp             1.1.2                h470a237_7    conda-forge
xorg-libxext              1.3.3                h470a237_4    conda-forge
xorg-libxfixes            5.0.3                h470a237_4    conda-forge
xorg-libxft               2.3.2                h7c24a56_3    conda-forge
xorg-libxmu               1.1.2                h470a237_2    conda-forge
xorg-libxpm               3.5.12               h470a237_2    conda-forge
xorg-libxrender           0.9.10               h470a237_2    conda-forge
xorg-libxt                1.1.5                h470a237_2    conda-forge
xorg-renderproto          0.11.1               h470a237_2    conda-forge
xorg-xextproto            7.3.0                h470a237_2    conda-forge
xorg-xproto               7.0.31               h470a237_7    conda-forge
xz                        5.2.4                h14c3975_4  
yaml                      0.1.7                had09818_2  
zlib                      1.2.11               ha838bed_2

Describe the bug
A clear and concise description of what the bug is.

The input data consistency checks do not catch the following user error: Running arrow with a BAM file mapped against referenceA.fasta but accidentally supplying referenceB.fasta, where the two references are different but have many contig names in common.

In this case I was using two different assemblies from Canu, which names its contigs like "tig0000000n" (not always counted continuously - many will be missing), resulting in many contig names in common.

Error message
Paste the error message / stack.

arrow -j 32 -r wrong_reference.fasta -o after_arrow.fasta -o after_arrow.vcf pbalign_merged.bam
[WARNING] Some reference contigs aligned against are not found in the reference FASTA.  Will process only those contigs supported by the reference FASTA.

The big problem is there was no error message - I want this to fail!

To Reproduce
Steps to reproduce the behavior. Providing a minimal test dataset on which we can reproduce the behavior will generally lead to quicker turnaround time!

Regrettably the files are too large to easily share, and are currently confidential.

Expected behavior
A clear and concise description of what you expected to happen.

Referring to the source code, the warning is from here:

https://github.com/PacificBiosciences/GenomicConsensus/blob/master/GenomicConsensus/reference.py#L124
https://github.com/PacificBiosciences/GenomicConsensus/blob/develop/GenomicConsensus/reference.py#L124

In the comments above you say:

    # Contigs in FASTA may disagree with those in cmp.h5 ref info
    # table, for instance if the FASTA has been edited.  Here's how we
    # handle things:
    #
    # |fastaContigs \   cmpContigs| > 0 : OK, extra FASTA contigs just ignored
    # |cmpContigs   \ fastaContigs| > 0 : Not necessarily OK---a warning should be
    #                                     issued.  We then proceed to operate on
    #                                     the contigs that are in both.
    # |cmpContigs ^ fastaContigs| == 0  : Nothing to work with.  This is an error.
    #
    # While we formerly used MD5s to vouch for the identity of a
    # contig, we now use the name.  This is an inferior approach but
    # is necessary, in using the FastaTable.

In my example, checking the MD5 would have revealed that while there were some contigs of the same name in the FASTA and BAM, they did not match.

The BAM file does contain the MD5 of each reference contig as part of the @SQ header lines, but also contain the expected length.

I can understand not checking the MD5 due to speed, but checking the name and lengths are consistent ought to be a very quick test and would catch most cases (even if by chance the first few contigs matched up, the chances of all the contig lengths matching ought to be minimal).

Falcon: too many jobs in daligner-runs folder, increases the space usage

Hi,

I have two plant genomes with similar genome size (1Gb), repeat content (~70%) and PacBio coverage (90-100X), and I used falcon (falcon-kit 1.2.3, from pb-assembly installed through conda) to assemble both. The pipeline was completed without any issues for only one of them. And the assembly is good. The other one was far more slower than the finished one and generated 1049 jobs in daligner-runs (occupying 11T) and 500 jobs (200 jobs occupy 4T) in las-merge-runs. I had to kill the jobs because the server was almost out off space. Then I added the repeat filter option in the configuration file, deleted the output files and started the pipeline again, I found I still have 1049 jobs in daligner-split/daligner-scripts/. I am running the job locally. Is there any thing I can do to reduce the job number? Here is my configure file:

[General]
job_type = local
input_fofn=input.fofn
input_type=raw
pa_DBdust_option=true
pa_fasta_filter_option=streamed-median

pa_HPCTANmask_option =
#pa_HPCREPmask_option =
pa_REPmask_code=0,300;0,300;0,300

#Data Partitioning   large genomes
pa_DBsplit_option=-x500 -s200
ovlp_DBsplit_option=-x500 -s200

#Pre-assembly
genome_size=923000000
seed_coverage=40
length_cutoff=-1
pa_HPCdaligner_option=-v -B128 -M24
pa_daligner_option=-e0.8 -l2000 -k18 -h256 -w8 -s100
falcon_sense_option=--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 300 --n_core 8
falcon_sense_greedy=False

#Pread overlapping
ovlp_daligner_option=-e.93 -k24 -l1800 -h600 -s100
ovlp_HPCdaligner_option=-v -B128 -M24

#Final Assembly
overlap_filtering_setting=--max-diff 100 --max-cov 300 --min-cov 4 --n_core 12
fc_ovlp_to_graph_option=
length_cutoff_pr=4000

# job concurrency settings for...
# all jobs
default_concurrent_jobs = 64
# preassembly
pa_concurrent_jobs = 64
# consensus calling of preads
cns_concurrent_jobs = 64
# overlap detection
ovlp_concurrent_jobs = 64

jobqueue = your_queue
sge_option_da = -pe smp 4 -q %(jobqueue)s
sge_option_la = -pe smp 20 -q %(jobqueue)s
sge_option_pda = -pe smp 6 -q %(jobqueue)s
sge_option_pla = -pe smp 16 -q %(jobqueue)s
sge_option_fc = -pe smp 24 -q %(jobqueue)s
sge_option_cns = -pe smp 8 -q %(jobqueue)s

no bam output after running isoseq primer removal

Dear Armin,
Thanks for developing isoseq3 tool to analyze the pacBio IsoSeq data.
I have two samples (wild type and knock out of a gene of interest) for which I generated pacBio isoSeq data to investigate alternative splicing.

For wt samples, the primer removal turns out to be good. However, for the knock out sample, I didn't get any bam output. Yet for the wild type sample, I got the bam output "wt.demux.primer_5p--primer_3p.bam".

The commands I ran for both samples are similar:
(1) For knock out sample:
lima -j 10 --isoseq --dump-clips --no-pbi ../1_ccs/ko.bam primers.fasta ko.demux.bam

(2) for wild type sample:
lima -j 10 --isoseq --dump-clips --no-pbi ../1_ccs/wt.bam primers.fasta wt.demux.bam

The summary output for wild type (wt.demux.lima.summary) is like below:
ZMWs input (A) : 240617
ZMWs above all thresholds (B) : 179301 (75%)
ZMWs below any threshold (C) : 61316 (25%)

ZMW marginals for (C):
Below min length : 1582 (3%)
Below min score : 0 (0%)
Below min end score : 36519 (60%)
Below min passes : 393 (1%)
Below min score lead : 0 (0%)
Below min ref span : 21377 (35%)
Without adapter : 393 (1%)
Undesired 5p--5p pairs : 13252 (22%)
Undesired 3p--3p pairs : 27381 (45%)
Undesired no hit : 393 (1%)

ZMWs for (B):
With different barcodes : 179301 (100%)
Coefficient of correlation : 0%

ZMWs for (A):
Allow diff barcode pair : 240224 (100%)
Allow same barcode pair : 240224 (100%)

Reads for (B):
Above length : 179301 (100%)
Below length : 0 (0%)

The output for knock out sample (ko.demux.lima.summary) is like below:
ZMWs input (A) : 44194
ZMWs above all thresholds (B) : 0 (0%)
ZMWs below any threshold (C) : 44194 (100%)

ZMW marginals for (C):
Below min length : 587 (1%)
Below min score : 0 (0%)
Below min end score : 43616 (99%)
Below min passes : 578 (1%)
Below min score lead : 0 (0%)
Below min ref span : 43559 (99%)
Without adapter : 578 (1%)
Undesired 5p--5p pairs : 22716 (51%)
Undesired 3p--3p pairs : 16185 (37%)
Undesired no hit : 578 (1%)

ZMWs for (B):
Coefficient of correlation : -nan%

ZMWs for (A):
Allow diff barcode pair : 43616 (99%)
Allow same barcode pair : 43616 (99%)

Reads for (B):
Above length : 0 (-nan%)
Below length : 0 (-nan%)

I noticed that the number of above length contigs in knockout is 0. Does this mean that it is a failed run in the knockout sample?

Also, I wonder what "reads for (A) and reads for (B) mean in the summary output.

As you are the expert who wrote the software, do you have an idea of what this means?

Many thanks for your time in reading this email.

I look forward to hearing from you.

Best,

Zhenzhen

[ERROR]Task Node(0-rawreads/build) failed with exit-code=1

Operating system
CentOS 6.5

Package name
pb-assembly

Describe the bug
PB-assembly on a very small E. coli dataset is failing with exit code 1 and some error messages.

Error message

falcon-kit 1.2.3
pypeflow 2.1.0
[INFO]Setup logging from file "None".
[INFO]$ lfs setstripe -c 12 /lustre1/gd98309/funzip/ecoli3 >
[WARNING]'lfs setstripe -c 12 /lustre1/gd98309/funzip/ecoli3' failed to produce any output.
[INFO]Lustre filesystem detected. This lfs stripe (12) should propagate to subdirs of '/lustre1/gd98309/funzip/eco
li3'.
[INFO]fc_run started with configuration fc_run.cfg
[WARNING]Unexpected keys in input config: set(['ovlp_HPCTANmask_option'])
[INFO]cfg=
{
  "General": {
    "LA4Falcon_preload": false,
    "avoid_text_file_busy": true,
    "bestn": 12,
    "dazcon": false,
    "falcon_sense_greedy": false,
    "falcon_sense_option": "--output-multi --min-idt 0.70 --min-cov 2 --max-n-read 400",
    "falcon_sense_skip_contained": false,
    "fc_ovlp_to_graph_option": " --min-len 1000",
    "genome_size": "4600000",
    "input_fofn": "input.fofn",
    "input_type": "raw",
    "length_cutoff": "-1",
    "length_cutoff_pr": "1000",
    "overlap_filtering_setting": "--max-diff 100 --max-cov 100 --min-cov 2",
    "ovlp_DBsplit_option": "-x500 -s200",
    "ovlp_HPCTANmask_option": "-l500",
    "ovlp_HPCdaligner_option": "-v -B128 -M24",
    "ovlp_daligner_option": "-e.93 -l2500 -k24 -h1024 -w6 -s100",
    "pa_DBdust_option": "True",
    "pa_DBsplit_option": "-x500 -s200",
    "pa_HPCTANmask_option": "",
    "pa_HPCdaligner_option": "-v -B128 -M24",
    "pa_REPmask_code": "0,300;0,300;0,300",
    "pa_daligner_option": "-e.8 -l2000 -k18 -h480  -w8 -s100",
    "pa_dazcon_option": "-j 4 -x -l 500",
    "pa_fasta_filter_option": "pass",
    "seed_coverage": "20",
    "skip_checks": true,
    "target": "assembly"
  },
  "job.defaults": {
    "JOB_QUEUE": "default",
    "MB": "32768",
    "NPROC": "8",
    "job_type": "local",
    "njobs": "32",
    "pwatcher_type": "blocking",
    "submit": "qsub -S /bin/bash -sync y -V  \\\n-q ${JOB_QUEUE}     \\\n-N ${JOB_NAME}      \\\n-o \"${JOB_STDOUT
}\"  \\\n-e \"${JOB_STDERR}\"  \\\n-pe smp ${NPROC}    \\\n-l h_vmem=${MB}M    \\\n\"${JOB_SCRIPT}\"",
    "use_tmpdir": false
  },
  "job.step.asm": {
    "MB": "32768",
    "NPROC": "4",
    "njobs": "3"
  },
  "job.step.cns": {
    "MB": "49152",
    "NPROC": "4",
    "njobs": "2"
  },
  "job.step.da": {},
  "job.step.la": {},
  "job.step.pda": {},
  "job.step.pla": {}
}
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.blocking' from '/home/gd98309/.conda/envs/falcond
a/lib/python2.7/site-packages/pwatcher/blocking.pyc'>
[INFO]job_type='local', (default)job_defaults={'JOB_QUEUE': 'default', 'pwatcher_type': 'blocking', 'use_tmpdir': 
False, 'MB': '32768', 'job_type': 'local', 'submit': 'qsub -S /bin/bash -sync y -V  \\\n-q ${JOB_QUEUE}     \\\n-N
 ${JOB_NAME}      \\\n-o "${JOB_STDOUT}"  \\\n-e "${JOB_STDERR}"  \\\n-pe smp ${NPROC}    \\\n-l h_vmem=${MB}M    
\\\n"${JOB_SCRIPT}"', 'NPROC': '8', 'njobs': '32'}, use_tmpdir=False, squash=False, job_name_style=0
[INFO]Setting max_jobs to 32; was None
[INFO]Num unsatisfied: 2, graph: 2
[INFO]About to submit: Node(0-rawreads/build)
[INFO]Popen: 'qsub -S /bin/bash -sync y -V  \
-q default     \
-N Pa93d51772c1627      \
-o "/lustre1/gd98309/funzip/ecoli3/0-rawreads/build/run-Pa93d51772c1627.bash.stdout"  \
-e "/lustre1/gd98309/funzip/ecoli3/0-rawreads/build/run-Pa93d51772c1627.bash.stderr"  \
-pe smp 1    \
-l h_vmem=4000M    \
"/home/gd98309/.conda/envs/falconda/lib/python2.7/site-packages/pwatcher/mains/job_start.sh"'
[INFO](slept for another 0.0s -- another 1 loop iterations)
qsub: script file 'y' cannot be loaded - No such file or directory
[ERROR]Task Node(0-rawreads/build) failed with exit-code=1
[ERROR]Some tasks are recently_done but not satisfied: set([Node(0-rawreads/build)])
[ERROR]ready: set([])
	submitted: set([])
[ERROR]Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
Traceback (most recent call last):
  File "/home/gd98309/.conda/envs/falconda/bin/fc_run", line 11, in <module>
    load_entry_point('falcon-kit==1.2.3', 'console_scripts', 'fc_run')()
  File "/home/gd98309/.conda/envs/falconda/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 726, in mai
n
    main1(argv[0], args.config, args.logger)
  File "/home/gd98309/.conda/envs/falconda/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 76, in main
1
    input_fofn_fn=input_fofn_fn,
  File "/home/gd98309/.conda/envs/falconda/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 242, in run
    dist=Dist(NPROC=4, MB=4000, job_dict=config['job.step.da']),
  File "/home/gd98309/.conda/envs/falconda/lib/python2.7/site-packages/falcon_kit/pype.py", line 106, in gen_paral
lel_tasks
    wf.refreshTargets()
  File "/home/gd98309/.conda/envs/falconda/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 2
77, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/home/gd98309/.conda/envs/falconda/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 3
61, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: set([Node(0-rawreads/build)])

To Reproduce

I am using this E. coli test dataset:
https://github.com/PacificBiosciences/DevNet/wiki/E.-coli-Bacterial-Assembly
I am only using one of the subreads fasta file.

Here is my script:

ml Miniconda3
source activate /home/gd98309/.conda/envs/falconda

cd $OUTDIR

fc_run fc_run.cfg &> falcon-$PBS_JOBID.log

source deactivate
ml unload Miniconda3

Isoseq3 lima to remove primers of Clontech library

Hi everyone,
Did someone try to use the lima to remove the primers for the sequencing data based on the Clontech library? This is one issue because the 5'primer and 3'primer are very similar (reverse complemented, and only 6 more bases for 5primer):
>primer_5p
AAGCAGTGGTATCAACGCAGAGTACATGGGG
>primer_3p
GTACTCTGCGTTGATACCACTGCT

The result for the orientation is not good, as a high percentage (42%) of CCS reads are either 5p--5p or 3p--3p, possibly because the program cannot well distinguish 5'primer and 3'primer. Does anyone know how to fix this issue? Thanks!

ZMWs input (A) : 600131
ZMWs above all thresholds (B) : 349504 (58%)
ZMWs below any threshold (C) : 250627 (42%)

ZMW marginals for (C):
Below min length : 468 (0%)
Below min score : 0 (0%)
Below min end score : 44890 (18%)
Below min passes : 205 (0%)
Below min score lead : 0 (0%)
Below min ref span : 37100 (15%)
Without adapter : 205 (0%)
Undesired 5p--5p pairs : 58600 (23%)
Undesired 3p--3p pairs : 174946 (70%)
Undesired no hit : 205 (0%)

ZMWs for (B):
With different barcodes : 349504 (100%)
Coefficient of correlation : 0%

ZMWs for (A):
Allow diff barcode pair : 599926 (100%)
Allow same barcode pair : 599926 (100%)

Reads for (B):
Above length : 349504 (100%)
Below length : 0 (0%)

isoseq3 cluster

Running isoseq3 v3.0.0 on Ubuntu 16.04 with conda as in this tutorial: https://github.com/PacificBiosciences/IsoSeq_SA3nUP/wiki/Tutorial:-Installing-and-Running-Iso-Seq-3-using-Conda

I am following the tutorial down to the letter and cannot get the isoseq3 cluster step to work, no matter what options I use. When I run it without arguments I get the help page, which leads me to believe it must be functional. But every time I put any input in, I get "Illegal instruction". I have the command as requested; "isoseq3 cluster demux.P5--P3.bam unpolished.bam", with and without extra options. Still no luck.

Does anyone know what the problem could be?

falcon_kit.mains.fasta_filter is crashing

Operating system
CentOS Linux release 7.3.1611

Package name
pb-assembly installed today via anaconda

Describe the bug
crashing command: python -m falcon_kit.mains.fasta_filter median -

Falcon crashes because falcon_kit.mains.fasta_filter does not accept file name input from the standard input, but expects a file path.
Following test call works: python -m falcon_kit.mains.fasta_filter median

Is falcon_kit 1.2.2 outdated?

Error message

#fc_fasta2fasta < my.input.fofn >| fc.fofn
while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter median - | fasta2DB -v raw_reads -i${fn##*/}; done < my.input.fofn
+ read fn
+ fasta2DB -v raw_reads -im170528_174820_42276_c101191972550000001823283109291706_s1_p0.1.subreads.fasta
+ python -m falcon_kit.mains.fasta_filter median -
+ cat /home/lehmanr/dataStore/melPacbio/r000112_42276_170525/DAM4_0245/A07_1/Analysis_Results/m170528_174820_42276_c101191972550000001823283109291706_s1_p0.1.subreads.fasta
falcon-kit 1.2.2
pypeflow 2.0.4
Traceback (most recent call last):
  File "/home/lehmanr/.conda/envs/denovo_asm/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/lehmanr/.conda/envs/denovo_asm/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/lehmanr/anaconda_lib/.conda/envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 234, in <module>
    main(sys.argv)          # pragma: no cover
  File "/home/lehmanr/anaconda_lib/.conda/envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 231, in main
    args.func(args)
  File "/home/lehmanr/anaconda_lib/.conda/envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 164, in cmd_run_median_filter
    with open(args.input_path, 'r') as fp_in:
IOError: [Errno 2] No such file or directory: '-'
cat: write error: Broken pipe
Standard input is empty, terminating!

Is falcon_unzip.mains.fetch_reads hard coded to use one CPU?

Operating system
CentOS Linux release 7.5.1804 (Core)

Package name
falcon-unzip 1.1.3
falcon-kit 1.2.3
pypeflow 2.1.0

Conda environment

asn1crypto                0.24.0                   py27_0  
avro                      1.8.0                    py27_0    bioconda
bcftools                  1.9                  h4da6232_0    bioconda
bedtools                  2.27.1               he941832_2    bioconda
blas                      1.0                         mkl  
blasr                     5.3.2                hac9d22c_3    bioconda
blasr_libcpp              5.3.1                hac9d22c_2    bioconda
bwa                       0.7.17               ha92aebf_3    bioconda
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.8.24            ha4d7672_0    conda-forge
cairo                     1.14.12              h276e583_5    conda-forge
certifi                   2018.8.24             py27_1001    conda-forge
cffi                      1.11.5           py27he75722e_1  
chardet                   3.0.4                    py27_1  
conda                     4.5.11                   py27_0    conda-forge
conda-env                 2.6.0                         1  
cryptography              2.3.1            py27hc365091_0  
curl                      7.61.0               h93b3f91_2    conda-forge
cython                    0.28.5           py27hfc679d8_0    conda-forge
decorator                 4.3.0                      py_0    conda-forge
enum34                    1.1.6                    py27_1  
fontconfig                2.13.1               h65d0f4c_0    conda-forge
freetype                  2.9.1                h6debe1e_4    conda-forge
future                    0.16.0                   py27_2    conda-forge
futures                   3.2.0                    py27_0  
genomicconsensus          2.3.2                    py27_1    bioconda
gettext                   0.19.8.1             h5e8e0c9_1    conda-forge
glib                      2.56.2               h464dc38_0    conda-forge
graphite2                 1.3.12               hfc679d8_1    conda-forge
gsl                       2.2.1                h0c605f7_3  
h5py                      2.8.0            py27hb794570_1    conda-forge
harfbuzz                  1.9.0                hee26f79_1    conda-forge
hdf5                      1.10.2               hc401514_2    conda-forge
hmmer                     3.1b2                         0    biocore
htslib                    1.7                           0    bioconda
icu                       58.2                 hfc679d8_0    conda-forge
idna                      2.7                      py27_0  
intel-openmp              2019.0                      118  
ipaddress                 1.0.22                   py27_0  
iso8601                   0.1.12                     py_1    conda-forge
jpeg                      9c                   h470a237_1    conda-forge
krb5                      1.14.6                        0    conda-forge
libdeflate                1.0                  h470a237_0    bioconda
libedit                   3.1.20170329         h6b74fdf_2  
libffi                    3.2.1                hd88cf55_4  
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libiconv                  1.15                 h470a237_3    conda-forge
libopenblas               0.2.20               h9ac9557_7  
libpng                    1.6.35               ha92aebf_2    conda-forge
libssh2                   1.8.0                h5b517e9_2    conda-forge
libstdcxx-ng              8.2.0                hdf63c60_1  
libtiff                   4.0.9                he6b73bb_2    conda-forge
libuuid                   2.32.1               h470a237_2    conda-forge
libxcb                    1.13                 h470a237_2    conda-forge
libxml2                   2.9.8                h422b904_5    conda-forge
linecache2                1.0.0                    py27_0    conda-forge
minimap2                  2.12                 ha92aebf_0    bioconda
mkl                       2019.0                      118  
mkl_fft                   1.0.6                    py27_0    conda-forge
mkl_random                1.0.1                    py27_0    conda-forge
mummer                    3.23                    pl526_6    bioconda
mummer4                   4.0.0beta2      pl526hfc679d8_3    bioconda
ncurses                   6.1                  hf484d3e_0  
networkx                  2.1                        py_1    conda-forge
nim-falcon                0.0.0                         0    bioconda
numpy                     1.15.1           py27h1d66e8a_0  
numpy-base                1.15.1           py27h81de0dd_0  
openssl                   1.0.2p               h470a237_0    conda-forge
pango                     1.40.14              he752989_2    conda-forge
pb-assembly               0.0.1                    py27_0    bioconda
pb-dazzler                0.0.0                h470a237_0    bioconda
pb-falcon                 0.2.3            py27ha92aebf_0    bioconda
pbalign                   0.3.1                    py27_0    bioconda
pbbam                     0.18.0               h1310cd9_1    bioconda
pbcommand                 1.1.1            py27h24bf2e0_1    bioconda
pbcore                    1.5.1                    py27_1    bioconda
pcre                      8.41                 hfc679d8_3    conda-forge
perl                      5.26.2               h470a237_0    conda-forge
pip                       10.0.1                   py27_0  
pixman                    0.34.0               h470a237_3    conda-forge
pthread-stubs             0.4                  h470a237_1    conda-forge
purge_haplotigs           1.0.1                hec630e7_1    mroachawri
pycosat                   0.6.3            py27h14c3975_0  
pycparser                 2.18                     py27_1  
pyopenssl                 18.0.0                   py27_0  
pysam                     0.14.1           py27hae42fb6_1    bioconda
pysocks                   1.6.8                    py27_0  
python                    2.7.15               h1571d57_0  
python-consensuscore      1.1.1            py27h02d93b8_1    bioconda
python-consensuscore2     3.1.0                    py27_1    bioconda
python-edlib              1.2.3            py27h470a237_1    bioconda
python-intervaltree       2.1.0                      py_0    bioconda
python-msgpack            0.5.6            py27h470a237_0    bioconda
python-sortedcontainers   2.0.4                      py_0    bioconda
pytz                      2018.5                     py_0    conda-forge
r-assertthat              0.2.0            r351h6115d3f_1    conda-forge
r-base                    3.5.1                h4fe35fd_0    conda-forge
r-cli                     1.0.0            r351h6115d3f_1    conda-forge
r-colorspace              1.3_2            r351hc070d10_2    conda-forge
r-crayon                  1.3.4            r351h6115d3f_1    conda-forge
r-digest                  0.6.16           r351hc070d10_1    conda-forge
r-fansi                   0.3.0            r351hc070d10_0    conda-forge
r-ggplot2                 3.0.0            r351h6115d3f_1    conda-forge
r-glue                    1.3.0            r351h470a237_2    conda-forge
r-gtable                  0.2.0            r351h6115d3f_1    conda-forge
r-labeling                0.3              r351h6115d3f_1    conda-forge
r-lattice                 0.20_35          r351hc070d10_0    conda-forge
r-lazyeval                0.2.1            r351hc070d10_2    conda-forge
r-magrittr                1.5              r351h6115d3f_1    conda-forge
r-mass                    7.3_50           r351hc070d10_2    conda-forge
r-matrix                  1.2_14           r351hc070d10_2    conda-forge
r-mgcv                    1.8_24           r351hc070d10_2    conda-forge
r-munsell                 0.5.0            r351h6115d3f_1    conda-forge
r-nlme                    3.1_137          r351h364d78e_0    conda-forge
r-pillar                  1.3.0            r351h6115d3f_0    conda-forge
r-plyr                    1.8.4            r351h9d2a408_2    conda-forge
r-r6                      2.2.2            r351h6115d3f_1    conda-forge
r-rcolorbrewer            1.1_2            r351h6115d3f_1    conda-forge
r-rcpp                    0.12.17          r351h9d2a408_2    conda-forge
r-reshape2                1.4.3            r351h9d2a408_2    conda-forge
r-rlang                   0.2.1            r351h470a237_2    conda-forge
r-scales                  1.0.0            r351h9d2a408_1    conda-forge
r-stringi                 1.2.4            r351h9d2a408_0    conda-forge
r-stringr                 1.3.1            r351h6115d3f_1    conda-forge
r-tibble                  1.4.2            r351hc070d10_2    conda-forge
r-utf8                    1.1.4            r351hc070d10_0    conda-forge
r-viridislite             0.3.0            r351h6115d3f_1    conda-forge
r-withr                   2.1.2            r351h6115d3f_0    conda-forge
readline                  7.0                  h7b6447c_5  
requests                  2.19.1                   py27_0  
ruamel_yaml               0.15.46          py27h14c3975_0  
samtools                  1.9                  h8ee4bcc_1    bioconda
setuptools                40.2.0                   py27_0  
six                       1.11.0                   py27_1  
sqlite                    3.24.0               h84994c4_0  
tk                        8.6.8                hbc83047_0  
traceback2                1.4.0                    py27_0    conda-forge
unittest2                 1.1.0                      py_0    conda-forge
urllib3                   1.23                     py27_0  
wheel                     0.31.1                   py27_0  
xorg-kbproto              1.0.7                h470a237_2    conda-forge
xorg-libice               1.0.9                h470a237_4    conda-forge
xorg-libsm                1.2.2                h8c8a85c_6    conda-forge
xorg-libx11               1.6.6                h470a237_0    conda-forge
xorg-libxau               1.0.8                h470a237_6    conda-forge
xorg-libxdmcp             1.1.2                h470a237_7    conda-forge
xorg-libxext              1.3.3                h470a237_4    conda-forge
xorg-libxrender           0.9.10               h470a237_2    conda-forge
xorg-libxt                1.1.5                h470a237_2    conda-forge
xorg-renderproto          0.11.1               h470a237_2    conda-forge
xorg-xextproto            7.3.0                h470a237_2    conda-forge
xorg-xproto               7.0.31               h470a237_7    conda-forge
xz                        5.2.4                h470a237_1    conda-forge
yaml                      0.1.7                had09818_2  
zlib                      1.2.11               ha838bed_2

Describe the bug
Apparently falcon_unzip.mains.fetch_reads is only using one CPU. Is this the expected behavior?

variantCaller Error

I receive this error while using Arrow.

Arrow: unsupported chemistries found: (unknown)
Arrow: unsupported chemistries found: (unknown)
Traceback (most recent call last):
  File "/home/fc464/lib/python2.7/site-packages/pbcommand-0.6.12-py2.7.egg/pbcommand/cli/core.py", line 137, in _pacbio_main_runner
    return_code = exe_main_func(*args, **kwargs)
  File "/home/fc464/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/main.py", line 340, in args_runner
    return tr.main()
  File "/home/fc464/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/main.py", line 270, in main
    self._configureAlgorithm(options, peekFile)
  File "/home/fc464/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/main.py", line 164, in _configureAlgorithm
    self._algorithmConfiguration = self._algorithm.configure(options, alnFile)
  File "/home/fc464/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/arrow/arrow.py", line 236, in configure
    die("Arrow: unsupported chemistries found: ({0})".format(", ".join(sorted(unsupp))))
  File "/home/fc464/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/utils.py", line 16, in die
    raise DieException(msg)
DieException: Arrow: unsupported chemistries found: (unknown)
[ERROR] Arrow: unsupported chemistries found: (unknown)
Traceback (most recent call last):
  File "/home/fc464/lib/python2.7/site-packages/pbcommand-0.6.12-py2.7.egg/pbcommand/cli/core.py", line 137, in _pacbio_main_runner
    return_code = exe_main_func(*args, **kwargs)
  File "/home/fc464/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/main.py", line 340, in args_runner
    return tr.main()
  File "/home/fc464/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/main.py", line 270, in main
    self._configureAlgorithm(options, peekFile)
  File "/home/fc464/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/main.py", line 164, in _configureAlgorithm
    self._algorithmConfiguration = self._algorithm.configure(options, alnFile)
  File "/home/fc464/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/arrow/arrow.py", line 236, in configure
    die("Arrow: unsupported chemistries found: ({0})".format(", ".join(sorted(unsupp))))
  File "/home/fc464/.local/lib/python2.7/site-packages/GenomicConsensus-2.3.2-py2.7.egg/GenomicConsensus/utils.py", line 16, in die
    raise DieException(msg)
DieException: Arrow: unsupported chemistries found: (unknown)

polish step fails in isoseq3 workflow

I installed Isoseq as in https://github.com/PacificBiosciences/IsoSeq_SA3nUP/wiki/Tutorial:-Installing-and-Running-Iso-Seq-3-using-Conda#running and ran all commands from that page and from https://github.com/PacificBiosciences/IsoSeq3 which give results until I try to polish.

The polishing command uses the unpolished.bam and the original subreads used for CCS but returns empty content in all polished. files as well as 3 warning during runtime.

isoseq3 polish unpolished.bam m54094_180927_125111.subreads.bam polished.bam
>|> 20181009 11:07:04.535 -|- WARN       -|- operator() -|- 0x7f917efcc700|| -|- No subreads for cluster transcript/828 in window 0-500
>|> 20181009 11:07:12.090 -|- WARN       -|- operator() -|- 0x7f9174fb8700|| -|- No subreads for cluster transcript/816 in window 0-500
>|> 20181009 11:07:12.158 -|- WARN       -|- operator() -|- 0x7f9174fb8700|| -|- No subreads for cluster transcript/816 in window 500-1000

The polished.bam has a header where you can see the previous commands reported.

samtools view -h polished.bam
@HD     VN:1.5  SO:unknown      pb:3.0.1
@RG     ID:e4927d21     PL:PACBIO       DS:READTYPE=TRANSCRIPT  PU:transcript   PM:SEQUEL
@PG     ID:ccs-3.0.0    PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-23616117   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-323CEACF   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-35184AC6   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-364747FC   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-3751DD58   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-37DCFA6C   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-3872F55A   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-39AE5771   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-39D589C3   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-47D43D26   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-4A82D30B   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-4CEBA27A   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-4D70490A   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-4DB78682   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-4E31A573   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-4FD8F07    PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-53693C07   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-5BD44F34   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-625B3E8A   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-66600F9A   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-6BC74ED3   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-73D12D9F   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:ccs-3.0.0-7CD52484   PN:ccs  VN:3.0.0        DS:Generate circular consensus sequences (ccs) from subreads.   CL:ccs 
@PG     ID:lima VN:1.8.0 (commit v1.8.0)        CL:lima --isoseq --dump-clips --no-pbi -j 24 merged_ccs.bam primers.fasta demux.bam
@PG     ID:sierra       VN:3.0.0 (commit v3.0.0)        CL:cluster demux.5p--3p.bam unpolished.bam -j 48
@PG     ID:tango        VN:0.7.1 (commit v0.4.0-121-g22a3096*)  CL:polish unpolished.bam m54094_180927_125111.subreads.bam polished.bam

-rw-r--r--  1 u0002316 domain users  606 Oct  9 13:20 polished.bam
-rw-r--r--  1 u0002316 domain users   65 Oct  9 13:20 polished.bam.pbi
-rw-r--r--  1 u0002316 domain users   20 Oct  9 13:20 polished.hq.fasta.gz
-rw-r--r--  1 u0002316 domain users   20 Oct  9 13:20 polished.hq.fastq.gz
-rw-r--r--  1 u0002316 domain users   20 Oct  9 13:20 polished.lq.fasta.gz
-rw-r--r--  1 u0002316 domain users   20 Oct  9 13:20 polished.lq.fastq.gz
-rw-r--r--  1 u0002316 domain users 1.3K Oct  9 13:20 polished.transcriptset.xml

The unpolished.bam is valid and contains 3901 sequences and the subreads are plenty as well.

Any idea what this could be coming from?

Thanks

Failure in generate_haplotigs_for_ctg((u'000519F', u'../../0-phasing/000519F/uow-00/proto', './uow-000519F', u'../../..', False))

executable=${PYPEFLOW_JOB_START_SCRIPT}
+ executable=/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F/run-Pda9cd060b80965.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
+ timeout=60

# Wait up to timeout seconds for the executable to become "executable",
# then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
    if [[ "${timeout}" == "0" ]]; then
        echo "timed out waiting for (${executable})"
        exit 77
    fi
    echo "not executable: '${executable}', waiting ${timeout}s"
    sleep 1
    timeout=$((timeout-1))
done
+ [[ ! -x /public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F/run-Pda9cd060b80965.bash ]]

/bin/bash ${executable}
+ /bin/bash /public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F/run-Pda9cd060b80965.bash
+ '[' '!' -d /public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F ']'
+ cd /public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F
+ eval '/bin/bash run.sh'
++ /bin/bash run.sh
export PATH=$PATH:/bin
+ export PATH=/home/mskkk/.pyenv/versions/anaconda-2.0.0/bin:/home/mskkk/.pyenv/libexec:/home/mskkk/.pyenv/plugins/python-build/bin:/home/mskkk/.pyenv/shims:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/home/mskkk/tools/RepeatMasker:/home/mskkk/tools/GARM_v0.7.5:/home/mskkk/tools/RepeatScout-1:/home/mskkk/tools/tophat-2.1.0.Linux_x86_64:/home/mskkk/tools/cufflinks-2.2.1.Linux_x86_64:/home/mskkk/tools/RECON1.05/bin:/home/mskkk/tools/biocode/lib:/home/mskkk/tools/RepeatModeler:/home/mskkk/tools/pigz-2.3.3:/home/mskkk/tools/canu-1.6/Linux-amd64/bin:/home/mskkk/tools/mugsy_x86-64-v1r2.2:/home/mskkk/tools/PhaME/bin:/home/mskkk/tools/paml4.8/src:/home/mskkk/tools/angsd:/home/mskkk/tools/mpj/bin:/home/mskkk/tools/hmmer-3.1b2-linux-intel-x86_64/binaries:/home/mskkk/tools/PILER/home/mskkk/tools/OMA/OMA/bin:/home/mskkk/tools/BRAKER1:/home/mskkk/tools/PhiPack/src/home/mskkk/toolsAxML:/home/mskkk/tools/phylip-3.696/exe:/home/mskkk/tools/bowtie2-2.2.5/:/home/mskkk/tools/geneid/bin:/home/mskkk/tools/wise-2.4.1/src/bin:/home/mskkk/tools/CEGMA_v2.5/bin:/home/mskkk/tools/bwa-0.7.12:/home/mskkk/smrtanalysis/install/smrtanalysis_2.3.0.140936/smrtcmds/bin:/home/mskkk/smrtanalysis/install/smrtanalysis_2.3.0.140936/analysis/bin:/home/mskkk/tools/MEGA7:/home/mskkk/tools/SPAdes-3.9.1-Linux/bin:/home/mskkk/tools/apache-maven-3.3.9/bin:/home/mskkk/tools/MetaCRAST/bin:/home/mskkk/tools/ClonalFrameML/src:/home/mskkk/toolsAxML:/home/mskkk/bin:/home/mskkk/tools/picard/dist:/home/mskkk/tools/PBSuite_15.8.24//bin:/home/mskkk/tools/picard-tools-2.1.0:/home/mskkk/tools/art:/home/mskkk/tools/soap.coverage:/home/mskkk/tools/soap2.21release:/home/mskkk/tools/NxTrim:/home/mskkk/tools/circos-0.69/bin:/home/mskkk/tools/anaconda2/bin:/home/mskkk/tools/bbmap:/home/mskkk/tools/OMA/OMA/bin:/home/mskkk/tools/pIRS_111:/home/mskkk/.pyenv/bin:/home/mskkk/toolsoary/bin:/usr/local/gcc-5.4/bin:/home/mskkk/toolsacon/tools/graphmap/bin/Linux-x64:/home/mskkk/toolsacon/tools/minimap:/home/mskkk/toolsacon/tools/edlib/src:/home/mskkk/toolsacon/scripts:/home/mskkk/toolsacon/bin:/home/mskkk/tools/miniasm:/home/mskkk/tools/pitchfork/deployment/bin:/home/mskkk/tools/DBG2OLC:/home/mskkk/tools/quickmerge/merger:/home/mskkk/tools/MaSuRCA-3.2.2_RC1/bin:/home/mskkk/tools/MECAT/Linux-amd64/bin:/home/mskkk/tools/sratoolkit.2.8.2-1-centos_linux64/bin:/home/mskkk/tools/wtdbg:/home/mskkk/tools/HaploMerger2_20161205/bin:/home/mskkk/tools/HaploMerger2_20161205/chainNet_jksrc20100603_centOS6:/home/mskkk/bin/x86_64-redhat-linux-gnu:/home/mskkk/tools/DEXTRACTOR:/home/mskkk/hhf/soft/MUMmer3.23/:/home/mskkk/hhf/soft/centrifuge-1.0.3-beta:/home/mskkk/hhf/soft/ANIcalculator_v1:/home/mskkk/hhf/soft/pplacer-Linux-v1.1.alpha19:/home/mskkk/hhf/soft/FragGeneScan1.30:/home/mskkk/tools/bin:/home/mskkk/tools/mash-Linux64-v1.1.1:/home/mskkk/hhf/soft/minced:/home/mskkk/tools/pitchfork/deployment/bin:/home/mskkk/tools/smrtanalysis/install/smrtlink-release_5.0.1.9585/bundles/smrttools/install/smrttools-release_5.0.1.9578/smrtcmds/bin:/home/mskkk/hhf/soft:/home/mskkk/tools/prokka-1.11/bin:/home/mskkk/tools/FastME-master-09a5862b5a227724b0d034dc33b6f1bf802cfb99/src:/home/mskkk/miniconda3/bin:/home/mskkk/tools/minimap:/home/mskkk/tools/signalp-4.1:/home/mskkk/tools/Roary/bin:/home/mskkk/msq/homer/bin:/home/mskkk/test2/test2/blastx/primer/primer3-2.4.0/src:/home/mskkk/tools/snippy/bin:/home/mskkk/tools/autoANI/scripts/edirect:/home/mskkk/tools/pplacer-Darwin-v1.1.alpha17-6-g5cecf99:/home/mskkk/tools/harvesttools-Linux64-v1.2:/home/mskkk/tools/pplacer-Linux-v1.1.alpha17:/home/mskkk/tools/cdhit:/home/mskkk/tools/nim-0.18.0/bin:/home/mskkk/tools/Roary/bin:/home/mskkk/tools/FALCON-Phase/bin:/home/mskkk/.local/bin:/home/mskkk/bin:/home/mskkk/msq/HI-C/tools/samtools-0.1.19:/home/mskkk/tools/maker/bin:/home/mskkk/hhf/soft/Mash/bin:/home/mskkk/hhf/soft/ANIcalculator_v1:/bin
+ PATH=/home/mskkk/.pyenv/versions/anaconda-2.0.0/bin:/home/mskkk/.pyenv/libexec:/home/mskkk/.pyenv/plugins/python-build/bin:/home/mskkk/.pyenv/shims:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/home/mskkk/tools/RepeatMasker:/home/mskkk/tools/GARM_v0.7.5:/home/mskkk/tools/RepeatScout-1:/home/mskkk/tools/tophat-2.1.0.Linux_x86_64:/home/mskkk/tools/cufflinks-2.2.1.Linux_x86_64:/home/mskkk/tools/RECON1.05/bin:/home/mskkk/tools/biocode/lib:/home/mskkk/tools/RepeatModeler:/home/mskkk/tools/pigz-2.3.3:/home/mskkk/tools/canu-1.6/Linux-amd64/bin:/home/mskkk/tools/mugsy_x86-64-v1r2.2:/home/mskkk/tools/PhaME/bin:/home/mskkk/tools/paml4.8/src:/home/mskkk/tools/angsd:/home/mskkk/tools/mpj/bin:/home/mskkk/tools/hmmer-3.1b2-linux-intel-x86_64/binaries:/home/mskkk/tools/PILER/home/mskkk/tools/OMA/OMA/bin:/home/mskkk/tools/BRAKER1:/home/mskkk/tools/PhiPack/src/home/mskkk/toolsAxML:/home/mskkk/tools/phylip-3.696/exe:/home/mskkk/tools/bowtie2-2.2.5/:/home/mskkk/tools/geneid/bin:/home/mskkk/tools/wise-2.4.1/src/bin:/home/mskkk/tools/CEGMA_v2.5/bin:/home/mskkk/tools/bwa-0.7.12:/home/mskkk/smrtanalysis/install/smrtanalysis_2.3.0.140936/smrtcmds/bin:/home/mskkk/smrtanalysis/install/smrtanalysis_2.3.0.140936/analysis/bin:/home/mskkk/tools/MEGA7:/home/mskkk/tools/SPAdes-3.9.1-Linux/bin:/home/mskkk/tools/apache-maven-3.3.9/bin:/home/mskkk/tools/MetaCRAST/bin:/home/mskkk/tools/ClonalFrameML/src:/home/mskkk/toolsAxML:/home/mskkk/bin:/home/mskkk/tools/picard/dist:/home/mskkk/tools/PBSuite_15.8.24//bin:/home/mskkk/tools/picard-tools-2.1.0:/home/mskkk/tools/art:/home/mskkk/tools/soap.coverage:/home/mskkk/tools/soap2.21release:/home/mskkk/tools/NxTrim:/home/mskkk/tools/circos-0.69/bin:/home/mskkk/tools/anaconda2/bin:/home/mskkk/tools/bbmap:/home/mskkk/tools/OMA/OMA/bin:/home/mskkk/tools/pIRS_111:/home/mskkk/.pyenv/bin:/home/mskkk/toolsoary/bin:/usr/local/gcc-5.4/bin:/home/mskkk/toolsacon/tools/graphmap/bin/Linux-x64:/home/mskkk/toolsacon/tools/minimap:/home/mskkk/toolsacon/tools/edlib/src:/home/mskkk/toolsacon/scripts:/home/mskkk/toolsacon/bin:/home/mskkk/tools/miniasm:/home/mskkk/tools/pitchfork/deployment/bin:/home/mskkk/tools/DBG2OLC:/home/mskkk/tools/quickmerge/merger:/home/mskkk/tools/MaSuRCA-3.2.2_RC1/bin:/home/mskkk/tools/MECAT/Linux-amd64/bin:/home/mskkk/tools/sratoolkit.2.8.2-1-centos_linux64/bin:/home/mskkk/tools/wtdbg:/home/mskkk/tools/HaploMerger2_20161205/bin:/home/mskkk/tools/HaploMerger2_20161205/chainNet_jksrc20100603_centOS6:/home/mskkk/bin/x86_64-redhat-linux-gnu:/home/mskkk/tools/DEXTRACTOR:/home/mskkk/hhf/soft/MUMmer3.23/:/home/mskkk/hhf/soft/centrifuge-1.0.3-beta:/home/mskkk/hhf/soft/ANIcalculator_v1:/home/mskkk/hhf/soft/pplacer-Linux-v1.1.alpha19:/home/mskkk/hhf/soft/FragGeneScan1.30:/home/mskkk/tools/bin:/home/mskkk/tools/mash-Linux64-v1.1.1:/home/mskkk/hhf/soft/minced:/home/mskkk/tools/pitchfork/deployment/bin:/home/mskkk/tools/smrtanalysis/install/smrtlink-release_5.0.1.9585/bundles/smrttools/install/smrttools-release_5.0.1.9578/smrtcmds/bin:/home/mskkk/hhf/soft:/home/mskkk/tools/prokka-1.11/bin:/home/mskkk/tools/FastME-master-09a5862b5a227724b0d034dc33b6f1bf802cfb99/src:/home/mskkk/miniconda3/bin:/home/mskkk/tools/minimap:/home/mskkk/tools/signalp-4.1:/home/mskkk/tools/Roary/bin:/home/mskkk/msq/homer/bin:/home/mskkk/test2/test2/blastx/primer/primer3-2.4.0/src:/home/mskkk/tools/snippy/bin:/home/mskkk/tools/autoANI/scripts/edirect:/home/mskkk/tools/pplacer-Darwin-v1.1.alpha17-6-g5cecf99:/home/mskkk/tools/harvesttools-Linux64-v1.2:/home/mskkk/tools/pplacer-Linux-v1.1.alpha17:/home/mskkk/tools/cdhit:/home/mskkk/tools/nim-0.18.0/bin:/home/mskkk/tools/Roary/bin:/home/mskkk/tools/FALCON-Phase/bin:/home/mskkk/.local/bin:/home/mskkk/bin:/home/mskkk/msq/HI-C/tools/samtools-0.1.19:/home/mskkk/tools/maker/bin:/home/mskkk/hhf/soft/Mash/bin:/home/mskkk/hhf/soft/ANIcalculator_v1:/bin
cd /public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F
+ cd /public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F
/bin/bash task.sh
+ /bin/bash task.sh
pypeflow 2.1.0
2018-09-30 09:11:08,309 - root - DEBUG - Running "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/pypeflow/do_task.py /public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F/task.json"
2018-09-30 09:11:08,310 - root - DEBUG - Checking existence of '/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F/task.json' with timeout=30
2018-09-30 09:11:08,310 - root - DEBUG - Loading JSON from '/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F/task.json'
2018-09-30 09:11:08,311 - root - DEBUG - {u'bash_template_fn': u'template.sh',
 u'inputs': {u'bash_template': u'../split/dummy.sh',
             u'units_of_work': u'../chunks/chunk_000519F/some-units-of-work.json'},
 u'outputs': {u'results': u'result-list.json'},
 u'parameters': {u'pypeflow_mb': u'131072', u'pypeflow_nproc': u'16'}}
2018-09-30 09:11:08,311 - root - WARNING - CD: '/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F' <- '/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F'
2018-09-30 09:11:08,311 - root - DEBUG - Checking existence of u'../chunks/chunk_000519F/some-units-of-work.json' with timeout=30
2018-09-30 09:11:08,311 - root - DEBUG - Checking existence of u'../split/dummy.sh' with timeout=30
2018-09-30 09:11:08,311 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2018-09-30 09:11:08,311 - root - WARNING - CD: '/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F' <- '/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F'
2018-09-30 09:11:08,312 - root - INFO - $('/bin/bash user_script.sh')
hostname
+ hostname
pwd
+ pwd
date
+ date
# Substitution will be similar to snakemake "shell".
    python -m falcon_unzip.mains.graphs_to_h_tigs_2 apply --units-of-work-fn=../chunks/chunk_000519F/some-units-of-work.json --results-fn=result-list.json
+ python -m falcon_unzip.mains.graphs_to_h_tigs_2 apply --units-of-work-fn=../chunks/chunk_000519F/some-units-of-work.json --results-fn=result-list.json

falcon-unzip 1.1.3
falcon-kit 1.2.3
pypeflow 2.1.0
[INFO 2018-09-30 09:11:09] Loading units-of-work from '../chunks/chunk_000519F/some-units-of-work.json'
[INFO 2018-09-30 09:11:09] Loading p assembly graph.
[INFO 2018-09-30 09:11:09] Counting 22,425,424 bytes from
  "../../../2-asm-falcon/sg_edges_list"
[INFO 2018-09-30 09:11:09]         #1 count=             61   0.00% 
[INFO 2018-09-30 09:11:09]         #3 count=            182   0.00% 
[INFO 2018-09-30 09:11:09]         #6 count=            363   0.00% 
[INFO 2018-09-30 09:11:09]        #10 count=            607   0.00% 
[INFO 2018-09-30 09:11:09]        #19 count=          1,153   0.01% 
[INFO 2018-09-30 09:11:09]        #36 count=          2,188   0.01% 
[INFO 2018-09-30 09:11:09]        #69 count=          4,190   0.02% 
[INFO 2018-09-30 09:11:09]       #134 count=          8,133   0.04% 
[INFO 2018-09-30 09:11:09]       #263 count=         15,967   0.07% 
[INFO 2018-09-30 09:11:09]       #521 count=         31,636   0.14% 
[INFO 2018-09-30 09:11:09]     #1,036 count=         62,921   0.28% 
[INFO 2018-09-30 09:11:09]     #2,065 count=        125,438   0.56% 
[INFO 2018-09-30 09:11:09]     #4,122 count=        250,399   1.12% 
[INFO 2018-09-30 09:11:09]     #8,236 count=        500,312   2.23% 
[INFO 2018-09-30 09:11:09]    #16,463 count=      1,000,061   4.46% 
[INFO 2018-09-30 09:11:09]    #32,914 count=      1,999,516   8.92% 
[INFO 2018-09-30 09:11:09]    #65,819 count=      3,998,394  17.83% 
[INFO 2018-09-30 09:11:09]   #102,734 count=      6,240,954  27.83% 
[INFO 2018-09-30 09:11:10]   #139,650 count=      8,483,539  37.83% 
[INFO 2018-09-30 09:11:10]   #176,567 count=     10,726,131  47.83% 
[INFO 2018-09-30 09:11:10]   #213,484 count=     12,968,692  57.83% 
[INFO 2018-09-30 09:11:11]   #250,399 count=     15,211,263  67.83% 
[INFO 2018-09-30 09:11:11]   #287,318 count=     17,453,864  77.83% 
[INFO 2018-09-30 09:11:11]   #324,233 count=     19,696,413  87.83% 
[INFO 2018-09-30 09:11:11]   #361,149 count=     21,938,970  97.83% 
[INFO 2018-09-30 09:11:12]   #369,156 count=     22,425,424 100.00% 
[INFO 2018-09-30 09:11:12] Counting 2,915,868 bytes from
  "../../../2-asm-falcon/utg_data"
[INFO 2018-09-30 09:11:12]         #1 count=             81   0.00% 
[INFO 2018-09-30 09:11:12]         #2 count=            236   0.01% 
[INFO 2018-09-30 09:11:12]         #4 count=            457   0.02% 
[INFO 2018-09-30 09:11:12]         #7 count=            813   0.03% 
[INFO 2018-09-30 09:11:12]        #14 count=          1,493   0.05% 
[INFO 2018-09-30 09:11:12]        #25 count=          2,835   0.10% 
[INFO 2018-09-30 09:11:12]        #49 count=          6,040   0.21% 
[INFO 2018-09-30 09:11:12]        #89 count=         11,320   0.39% 
[INFO 2018-09-30 09:11:12]       #172 count=         21,761   0.75% 
[INFO 2018-09-30 09:11:12]       #342 count=         42,574   1.46% 
[INFO 2018-09-30 09:11:12]       #679 count=         84,132   2.89% 
[INFO 2018-09-30 09:11:12]     #1,347 count=        167,122   5.73% 
[INFO 2018-09-30 09:11:12]     #2,636 count=        333,338  11.43% 
[INFO 2018-09-30 09:11:12]     #4,954 count=        624,928  21.43% 
[INFO 2018-09-30 09:11:12]     #7,242 count=        916,628  31.44% 
[INFO 2018-09-30 09:11:12]     #9,524 count=      1,208,363  41.44% 
[INFO 2018-09-30 09:11:12]    #11,839 count=      1,500,080  51.45% 
[INFO 2018-09-30 09:11:12]    #14,143 count=      1,791,737  61.45% 
[INFO 2018-09-30 09:11:12]    #16,456 count=      2,083,390  71.45% 
[INFO 2018-09-30 09:11:12]    #18,718 count=      2,375,127  81.46% 
[INFO 2018-09-30 09:11:12]    #21,037 count=      2,666,803  91.46% 
[INFO 2018-09-30 09:11:12]    #23,012 count=      2,915,868 100.00% 
[INFO 2018-09-30 09:11:12] Counting 346,781 bytes from
  "../../../2-asm-falcon/ctg_paths"
[INFO 2018-09-30 09:11:12]         #1 count=          4,008   1.16% 
[INFO 2018-09-30 09:11:12]         #3 count=         10,197   2.94% 
[INFO 2018-09-30 09:11:12]         #7 count=         19,380   5.59% 
[INFO 2018-09-30 09:11:12]        #14 count=         35,547  10.25% 
[INFO 2018-09-30 09:11:12]        #32 count=         68,262  19.68% 
[INFO 2018-09-30 09:11:12]        #56 count=        103,225  29.77% 
[INFO 2018-09-30 09:11:12]        #92 count=        138,720  40.00% 
[INFO 2018-09-30 09:11:12]       #145 count=        173,915  50.15% 
[INFO 2018-09-30 09:11:12]       #220 count=        208,670  60.17% 
[INFO 2018-09-30 09:11:12]       #347 count=        243,379  70.18% 
[INFO 2018-09-30 09:11:12]       #523 count=        278,145  80.21% 
[INFO 2018-09-30 09:11:12]       #777 count=        312,902  90.23% 
[INFO 2018-09-30 09:11:12]     #1,050 count=        346,781 100.00% 
[INFO 2018-09-30 09:11:14] Loading h assembly graph.
[INFO 2018-09-30 09:11:14] Counting 9,768,957 bytes from
  "../../1-hasm/sg_edges_list"
[INFO 2018-09-30 09:11:14]         #1 count=             60   0.00% 
[INFO 2018-09-30 09:11:14]         #2 count=            120   0.00% 
[INFO 2018-09-30 09:11:14]         #5 count=            295   0.00% 
[INFO 2018-09-30 09:11:14]        #10 count=            584   0.01% 
[INFO 2018-09-30 09:11:14]        #19 count=          1,099   0.01% 
[INFO 2018-09-30 09:11:14]        #36 count=          2,085   0.02% 
[INFO 2018-09-30 09:11:14]        #70 count=          4,050   0.04% 
[INFO 2018-09-30 09:11:14]       #136 count=          7,903   0.08% 
[INFO 2018-09-30 09:11:14]       #269 count=         15,616   0.16% 
[INFO 2018-09-30 09:11:14]       #534 count=         31,029   0.32% 
[INFO 2018-09-30 09:11:14]     #1,064 count=         61,764   0.63% 
[INFO 2018-09-30 09:11:14]     #2,124 count=        123,262   1.26% 
[INFO 2018-09-30 09:11:14]     #4,243 count=        246,174   2.52% 
[INFO 2018-09-30 09:11:14]     #8,478 count=        491,952   5.04% 
[INFO 2018-09-30 09:11:14]    #16,955 count=        983,481  10.07% 
[INFO 2018-09-30 09:11:14]    #33,798 count=      1,960,383  20.07% 
[INFO 2018-09-30 09:11:14]    #50,650 count=      2,937,328  30.07% 
[INFO 2018-09-30 09:11:14]    #67,488 count=      3,914,262  40.07% 
[INFO 2018-09-30 09:11:14]    #84,337 count=      4,891,187  50.07% 
[INFO 2018-09-30 09:11:14]   #101,186 count=      5,868,114  60.07% 
[INFO 2018-09-30 09:11:14]   #118,032 count=      6,845,037  70.07% 
[INFO 2018-09-30 09:11:15]   #134,888 count=      7,821,977  80.07% 
[INFO 2018-09-30 09:11:15]   #151,740 count=      8,798,917  90.07% 
[INFO 2018-09-30 09:11:15]   #168,478 count=      9,768,957 100.00% 
[INFO 2018-09-30 09:11:15] Counting 1,784,322 bytes from
  "../../1-hasm/utg_data"
[INFO 2018-09-30 09:11:15]         #1 count=            308   0.02% 
[INFO 2018-09-30 09:11:15]         #3 count=            667   0.04% 
[INFO 2018-09-30 09:11:15]        #10 count=          1,366   0.08% 
[INFO 2018-09-30 09:11:15]        #21 count=          2,600   0.15% 
[INFO 2018-09-30 09:11:15]        #42 count=          5,068   0.28% 
[INFO 2018-09-30 09:11:15]        #88 count=         10,108   0.57% 
[INFO 2018-09-30 09:11:15]       #178 count=         19,989   1.12% 
[INFO 2018-09-30 09:11:15]       #359 count=         39,795   2.23% 
[INFO 2018-09-30 09:11:15]       #713 count=         79,234   4.44% 
[INFO 2018-09-30 09:11:15]     #1,394 count=        158,125   8.86% 
[INFO 2018-09-30 09:11:15]     #2,776 count=        315,848  17.70% 
[INFO 2018-09-30 09:11:15]     #4,343 count=        494,324  27.70% 
[INFO 2018-09-30 09:11:15]     #5,943 count=        673,028  37.72% 
[INFO 2018-09-30 09:11:15]     #7,507 count=        851,475  47.72% 
[INFO 2018-09-30 09:11:15]     #9,066 count=      1,029,917  57.72% 
[INFO 2018-09-30 09:11:15]    #10,669 count=      1,208,376  67.72% 
[INFO 2018-09-30 09:11:15]    #12,224 count=      1,386,876  77.73% 
[INFO 2018-09-30 09:11:15]    #13,785 count=      1,565,543  87.74% 
[INFO 2018-09-30 09:11:15]    #15,336 count=      1,744,070  97.74% 
[INFO 2018-09-30 09:11:15]    #15,682 count=      1,784,322 100.00% 
[INFO 2018-09-30 09:11:15] Counting 849,964 bytes from
  "../../1-hasm/ctg_paths"
[INFO 2018-09-30 09:11:15]         #1 count=            370   0.04% 
[INFO 2018-09-30 09:11:15]         #2 count=            740   0.09% 
[INFO 2018-09-30 09:11:15]         #4 count=          1,550   0.18% 
[INFO 2018-09-30 09:11:15]        #10 count=          3,255   0.38% 
[INFO 2018-09-30 09:11:15]        #25 count=          6,359   0.75% 
[INFO 2018-09-30 09:11:15]        #53 count=         12,328   1.45% 
[INFO 2018-09-30 09:11:15]       #115 count=         24,406   2.87% 
[INFO 2018-09-30 09:11:15]       #241 count=         48,247   5.68% 
[INFO 2018-09-30 09:11:15]       #531 count=         95,668  11.26% 
[INFO 2018-09-30 09:11:15]     #1,116 count=        180,720  21.26% 
[INFO 2018-09-30 09:11:15]     #1,752 count=        265,757  31.27% 
[INFO 2018-09-30 09:11:15]     #2,408 count=        350,823  41.28% 
[INFO 2018-09-30 09:11:15]     #3,093 count=        435,930  51.29% 
[INFO 2018-09-30 09:11:15]     #3,795 count=        520,991  61.30% 
[INFO 2018-09-30 09:11:15]     #4,514 count=        606,002  71.30% 
[INFO 2018-09-30 09:11:15]     #5,243 count=        691,081  81.31% 
[INFO 2018-09-30 09:11:15]     #5,982 count=        776,120  91.31% 
[INFO 2018-09-30 09:11:15]     #6,628 count=        849,964 100.00% 
[INFO 2018-09-30 09:11:16] Loading phasing info and making the read ID sets.
[INFO 2018-09-30 09:11:16] Counting 9,072,742 bytes from
  "../../1-hasm/concatenated-rid-to-phase/rid_to_phase.all"
[INFO 2018-09-30 09:11:16]         #1 count=             23   0.00% 
[INFO 2018-09-30 09:11:16]         #2 count=             51   0.00% 
[INFO 2018-09-30 09:11:16]         #4 count=            102   0.00% 
[INFO 2018-09-30 09:11:16]         #8 count=            209   0.00% 
[INFO 2018-09-30 09:11:16]        #16 count=            413   0.00% 
[INFO 2018-09-30 09:11:16]        #31 count=            783   0.01% 
[INFO 2018-09-30 09:11:16]        #62 count=          1,531   0.02% 
[INFO 2018-09-30 09:11:16]       #123 count=          3,009   0.03% 
[INFO 2018-09-30 09:11:16]       #242 count=          5,961   0.07% 
[INFO 2018-09-30 09:11:16]       #478 count=         11,849   0.13% 
[INFO 2018-09-30 09:11:16]       #958 count=         23,629   0.26% 
[INFO 2018-09-30 09:11:16]     #1,914 count=         47,192   0.52% 
[INFO 2018-09-30 09:11:17]     #3,809 count=         94,312   1.04% 
[INFO 2018-09-30 09:11:17]     #7,589 count=        188,532   2.08% 
[INFO 2018-09-30 09:11:17]    #15,296 count=        376,964   4.15% 
[INFO 2018-09-30 09:11:17]    #30,389 count=        753,801   8.31% 
[INFO 2018-09-30 09:11:17]    #60,749 count=      1,507,466  16.62% 
[INFO 2018-09-30 09:11:17]    #96,621 count=      2,414,742  26.62% 
[INFO 2018-09-30 09:11:17]   #132,567 count=      3,322,035  36.62% 
[INFO 2018-09-30 09:11:18]   #168,859 count=      4,229,323  46.62% 
[INFO 2018-09-30 09:11:18]   #205,215 count=      5,136,606  56.62% 
[INFO 2018-09-30 09:11:18]   #241,435 count=      6,043,890  66.62% 
[INFO 2018-09-30 09:11:18]   #278,088 count=      6,951,184  76.62% 
[INFO 2018-09-30 09:11:18]   #314,652 count=      7,858,465  86.62% 
[INFO 2018-09-30 09:11:19]   #350,865 count=      8,765,754  96.62% 
[INFO 2018-09-30 09:11:19]   #362,896 count=      9,072,742 100.00% 
[INFO 2018-09-30 09:11:19] Counted 9,072,742 bytes in 362896 calls from:
  "../../1-hasm/concatenated-rid-to-phase/rid_to_phase.all"
[INFO 2018-09-30 09:11:19] Loading the 2-asm-falcon primary contigs.
[INFO 2018-09-30 09:11:19] Counting 220,086,479 bytes from
  "/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/2-asm-falcon/p_ctg.fa"
[INFO 2018-09-30 09:11:19]         #1 count=      5,737,008   2.61% 
[INFO 2018-09-30 09:11:19]         #3 count=     14,315,258   6.50% 
[INFO 2018-09-30 09:11:19]         #7 count=     28,971,807  13.16% 
[INFO 2018-09-30 09:11:19]        #14 count=     51,406,235  23.36% 
[INFO 2018-09-30 09:11:20]        #23 count=     74,898,079  34.03% 
[INFO 2018-09-30 09:11:20]        #34 count=     98,066,774  44.56% 
[INFO 2018-09-30 09:11:20]        #50 count=    120,371,811  54.69% 
[INFO 2018-09-30 09:11:20]        #72 count=    142,768,651  64.87% 
[INFO 2018-09-30 09:11:20]       #103 count=    164,996,177  74.97% 
[INFO 2018-09-30 09:11:20]       #167 count=    187,008,747  84.97% 
[INFO 2018-09-30 09:11:21]       #347 count=    209,047,237  94.98% 
[INFO 2018-09-30 09:11:21] Done loading 2-asm-falcon primary contigs.
[INFO 2018-09-30 09:11:21] Loading tiling paths.
[INFO 2018-09-30 09:11:21] Done loading tiling paths.
[INFO 2018-09-30 09:11:21] Loading the 1-hasm haplotigs.
[INFO 2018-09-30 09:11:22] Counting 254,918,040 bytes from
  "/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/1-hasm/p_ctg.fa"
[INFO 2018-09-30 09:11:22]         #1 count=        326,563   0.13% 
[INFO 2018-09-30 09:11:22]         #2 count=        664,713   0.26% 
[INFO 2018-09-30 09:11:22]         #5 count=      1,594,075   0.63% 
[INFO 2018-09-30 09:11:22]        #10 count=      2,961,522   1.16% 
[INFO 2018-09-30 09:11:22]        #21 count=      5,730,623   2.25% 
[INFO 2018-09-30 09:11:22]        #47 count=     11,040,295   4.33% 
[INFO 2018-09-30 09:11:22]       #106 count=     21,598,454   8.47% 
[INFO 2018-09-30 09:11:22]       #248 count=     42,625,133  16.72% 
[INFO 2018-09-30 09:11:22]       #458 count=     68,203,569  26.76% 
[INFO 2018-09-30 09:11:23]       #703 count=     93,774,277  36.79% 
[INFO 2018-09-30 09:11:23]       #981 count=    119,339,674  46.81% 
[INFO 2018-09-30 09:11:23]     #1,287 count=    144,886,940  56.84% 
[INFO 2018-09-30 09:11:23]     #1,631 count=    170,409,468  66.85% 
[INFO 2018-09-30 09:11:24]     #2,021 count=    195,907,420  76.85% 
[INFO 2018-09-30 09:11:24]     #2,482 count=    221,431,622  86.86% 
[INFO 2018-09-30 09:11:24]     #3,089 count=    246,937,442  96.87% 
[INFO 2018-09-30 09:11:24] Loading haplotigs.
[INFO 2018-09-30 09:11:24] Counting 3,314 units from
  "tiling_paths"
[INFO 2018-09-30 09:11:24]         #1 count=              1   0.03% 000040F-HAP000527F-000040F.1000002.0
[INFO 2018-09-30 09:11:24]         #3 count=              3   0.09% 000092F-HAP000333F-000092F.1000003.0
[INFO 2018-09-30 09:11:24]         #7 count=              7   0.21% 000021F-HAP000547F-000021F.3000004.0
[INFO 2018-09-30 09:11:24]        #15 count=             15   0.45% 000043F-HAP002339F-000043F.3000013.0
[INFO 2018-09-30 09:11:24]        #31 count=             31   0.94% 000008F-HAP002200F-000008F.3000008.1
[INFO 2018-09-30 09:11:24]        #63 count=             63   1.90% 000276F-HAP002571F-000276F.1000002.1
[INFO 2018-09-30 09:11:24]       #127 count=            127   3.83% 000142F-HAP003211F-000142F.3000001.1
[INFO 2018-09-30 09:11:24]       #255 count=            255   7.69% 000003F-HAP002644F-000003F.3000020.0
[INFO 2018-09-30 09:11:24]       #511 count=            511  15.42% 000006F-HAP000472F-000006F.1000014.0
[INFO 2018-09-30 09:11:24]       #842 count=            842  25.41% 000004F-HAP002552F-000004F.3000011.1
[INFO 2018-09-30 09:11:24]     #1,173 count=          1,173  35.40% 000019F-HAP001552F-000019F.1000015.1
[INFO 2018-09-30 09:11:24]     #1,504 count=          1,504  45.38% 000014F-HAP000808F-000014F.1000014.1
[INFO 2018-09-30 09:11:24]     #1,835 count=          1,835  55.37% 000004F-HAP001069F-000004F.5000022.0
[INFO 2018-09-30 09:11:24]     #2,166 count=          2,166  65.36% 000206F-HAP001399F-000206F.1000001.1
[INFO 2018-09-30 09:11:24]     #2,497 count=          2,497  75.35% 000049F-HAP002129F-000049F.3000006.0
[INFO 2018-09-30 09:11:24]     #2,828 count=          2,828  85.33% 000001F-HAP000445F-000001F.7000029.1
[INFO 2018-09-30 09:11:24]     #3,159 count=          3,159  95.32% 000034F-HAP000253F-000034F.1000006.0
[INFO 2018-09-30 09:11:24] Done loading haplotigs.
[INFO 2018-09-30 09:11:24] Loading sg_edges_list.
[INFO 2018-09-30 09:11:24] Counting 9,768,957 bytes from
  "../../1-hasm/sg_edges_list"
[INFO 2018-09-30 09:11:24]         #1 count=             60   0.00% 
[INFO 2018-09-30 09:11:24]         #2 count=            120   0.00% 
[INFO 2018-09-30 09:11:24]         #5 count=            295   0.00% 
[INFO 2018-09-30 09:11:24]        #10 count=            584   0.01% 
[INFO 2018-09-30 09:11:24]        #19 count=          1,099   0.01% 
[INFO 2018-09-30 09:11:24]        #36 count=          2,085   0.02% 
[INFO 2018-09-30 09:11:24]        #70 count=          4,050   0.04% 
[INFO 2018-09-30 09:11:24]       #136 count=          7,903   0.08% 
[INFO 2018-09-30 09:11:24]       #269 count=         15,616   0.16% 
[INFO 2018-09-30 09:11:24]       #534 count=         31,029   0.32% 
[INFO 2018-09-30 09:11:24]     #1,064 count=         61,764   0.63% 
[INFO 2018-09-30 09:11:25]     #2,124 count=        123,262   1.26% 
[INFO 2018-09-30 09:11:25]     #4,243 count=        246,174   2.52% 
[INFO 2018-09-30 09:11:25]     #8,478 count=        491,952   5.04% 
[INFO 2018-09-30 09:11:25]    #16,955 count=        983,481  10.07% 
[INFO 2018-09-30 09:11:25]    #33,798 count=      1,960,383  20.07% 
[INFO 2018-09-30 09:11:25]    #50,650 count=      2,937,328  30.07% 
[INFO 2018-09-30 09:11:25]    #67,488 count=      3,914,262  40.07% 
[INFO 2018-09-30 09:11:25]    #84,337 count=      4,891,187  50.07% 
[INFO 2018-09-30 09:11:25]   #101,186 count=      5,868,114  60.07% 
[INFO 2018-09-30 09:11:25]   #118,032 count=      6,845,037  70.07% 
[INFO 2018-09-30 09:11:25]   #134,888 count=      7,821,977  80.07% 
[INFO 2018-09-30 09:11:26]   #151,740 count=      8,798,917  90.07% 
[INFO 2018-09-30 09:11:26]   #168,478 count=      9,768,957 100.00% 
[INFO 2018-09-30 09:11:26] Counted 9,768,957 bytes in 168478 calls from:
  "../../1-hasm/sg_edges_list"
[INFO 2018-09-30 09:11:26] Done loading sg_edges_list.
[INFO 2018-09-30 09:11:26] Running 1 units of work.
[INFO 2018-09-30 09:11:26] UOW #0 of 1 ...
[INFO 2018-09-30 09:11:26] Entering generate_haplotigs_for_ctg(ctg_id=u'000519F', out_dir='./uow-000519F', base_dir=u'../../..'
[INFO 2018-09-30 09:11:26] New logging FileHandler: '/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F/uow-000519F/prototype.log'
[INFO 2018-09-30 09:11:26] Started processing contig: "000519F".
[INFO 2018-09-30 09:11:26] Fetching the p_ctg_seq.
[INFO 2018-09-30 09:11:26] Fetching the p_ctg_tiling_path.
[INFO 2018-09-30 09:11:26] Loading minced ctg seqs from u'../../0-phasing/000519F/uow-00/proto/minced.fasta' .
[INFO 2018-09-30 09:11:26] Counting 32,459 bytes from
  "/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/0-phasing/000519F/uow-00/proto/minced.fasta"
[INFO 2018-09-30 09:11:26]         #1 count=         32,458 100.00% 
[INFO 2018-09-30 09:11:26] Loading the phase relation graph from u'../../0-phasing/000519F/uow-00/proto/phase_relation_graph.gexf' .
[INFO 2018-09-30 09:11:26] Loading all regions from u'../../0-phasing/000519F/uow-00/proto/regions.json' .
[INFO 2018-09-30 09:11:26] Making bubble region list.
[INFO 2018-09-30 09:11:26] Retupling.
[INFO 2018-09-30 09:11:26] Assigning sequences to all regions.
[INFO 2018-09-30 09:11:26] Getting snp_haplotigs.
[INFO 2018-09-30 09:11:26] Writing haplotigs to disk: './uow-000519F/aln_snp_hasm_ctg.fasta'
[INFO 2018-09-30 09:11:26] [IS] Writing 1, h_name = 000519F-HAP003071F-000519F.1000001.1
[INFO 2018-09-30 09:11:26] [IS] Writing 2, h_name = 000519F-HAP002320F-000519F.1000001.0
[EXECUTE 2018-09-30 09:11:26] [2018/09/30 01:11:26] Executing "blasr --minMatch 15 --maxMatch 25 --advanceHalf --advanceExactMatches 10 --bestn 1 --nproc 16 --noSplitSubreads ./uow-000519F/aln_snp_hasm_ctg.fasta ../../../3-unzip/reads/000519F/ref.fa --sam --out ./uow-000519F/aln_snp_hasm_ctg.tmp.sam"
[INFO] 2018-09-30T09:11:26 [blasr] started.
[INFO] 2018-09-30T09:11:26 [blasr] ended.
[EXECUTE 2018-09-30 09:11:26] [2018/09/30 01:11:26]  Finished subprocess.
[EXECUTE 2018-09-30 09:11:26] [2018/09/30 01:11:26] Executing "samtools sort ./uow-000519F/aln_snp_hasm_ctg.tmp.sam -o ./uow-000519F/aln_snp_hasm_ctg.sam"
[EXECUTE 2018-09-30 09:11:26] [2018/09/30 01:11:26]  Finished subprocess.
[EXECUTE 2018-09-30 09:11:26] [2018/09/30 01:11:26] Executing "rm -f ./uow-000519F/aln_snp_hasm_ctg.tmp.sam"
[EXECUTE 2018-09-30 09:11:26] [2018/09/30 01:11:26]  Finished subprocess.
[INFO 2018-09-30 09:11:26] Loading the alignments.
[INFO 2018-09-30 09:11:26] Reorienting haplotigs.
[INFO 2018-09-30 09:11:26]   - qname = 000519F-HAP003071F-000519F.1000001.1
[INFO 2018-09-30 09:11:26] 
[INFO 2018-09-30 09:11:26]  start = (0, 642), end = (32435, 33061)
[INFO 2018-09-30 09:11:26] pos_of_interest for q_name: 000519F-HAP003071F-000519F.1000001.1
[INFO 2018-09-30 09:11:26] ((0, 642), (32435, 33061), '000519F-HAP003071F-000519F.1000001.1', 38200, '000519F', 32435, ('000519F', 1000001, 1))
[INFO 2018-09-30 09:11:26] 
[INFO 2018-09-30 09:11:26]  start = (0, 771), end = (32435, 33232)
[INFO 2018-09-30 09:11:26] pos_of_interest for q_name: 000519F-HAP002320F-000519F.1000001.0
[INFO 2018-09-30 09:11:26] ((0, 771), (32435, 33232), '000519F-HAP002320F-000519F.1000001.0', 39550, '000519F', 32435, ('000519F', 1000001, 0))
[INFO 2018-09-30 09:11:26] 
[INFO 2018-09-30 09:11:26] Function: "__main__"
[INFO 2018-09-30 09:11:26] len(sorted_bubble_regions) = 1
[INFO 2018-09-30 09:11:26] Handling prefix.
[INFO 2018-09-30 09:11:26] Entered function: "__main__"
[INFO 2018-09-30 09:11:26] Exiting function: "__main__"
[INFO 2018-09-30 09:11:26] Handling infix.
[INFO 2018-09-30 09:11:26] Handling suffix.
[INFO 2018-09-30 09:11:26] Entered function: "__main__"
[INFO 2018-09-30 09:11:26] Exiting function: "__main__"
[INFO 2018-09-30 09:11:26] Dunn!
[INFO 2018-09-30 09:11:26] Creating a haplotig graph.
[INFO 2018-09-30 09:11:26]   - Adding nodes.
[INFO 2018-09-30 09:11:26]     - region_id = 0, region_type = diploid, region_pos_start = 0, region_pos_end = 32435
[INFO 2018-09-30 09:11:26]       - [haplotig graph, adding node] key = 000519F-HAP003071F-000519F.1000001.1-0
[INFO 2018-09-30 09:11:26]       - [haplotig graph, adding node] key = 000519F-HAP002320F-000519F.1000001.0-0
[INFO 2018-09-30 09:11:26]   - Adding edges.
[INFO 2018-09-30 09:11:26]   - Hashing haplotigs.
[INFO 2018-09-30 09:11:26]   - Writing the haplotig graph in the gexf format.
[INFO 2018-09-30 09:11:26] Writing the haplotig_graph.gfa.
[INFO 2018-09-30 09:11:26]   - Writing all the haplotigs to disk in haplotigs.fasta.
[INFO 2018-09-30 09:11:26] Beginning to extract all p_ctg and h_ctg.
[INFO 2018-09-30 09:11:26] Extracting primary contig: p_ctg_id = 000519F
[INFO 2018-09-30 09:11:26] Making the haplotig segment coordinate relation lookup.
[INFO 2018-09-30 09:11:26] haplotig_segment_coords = {0: 0, 32435: 32419}

[INFO 2018-09-30 09:11:26] Extracting the associate haplotigs for p_ctg_id = 000519F
[ERROR 2018-09-30 09:11:26] Failure in generate_haplotigs_for_ctg((u'000519F', u'../../0-phasing/000519F/uow-00/proto', './uow-000519F', u'../../..', False))
Traceback (most recent call last):
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/falcon_unzip/mains/graphs_to_h_tigs_2.py", line 74, in run_generate_haplotigs_for_ctg
    return generate_haplotigs_for_ctg(ctg_id, allow_multiple_primaries, out_dir, unzip_dir, proto_dir, logger)
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/falcon_unzip/mains/graphs_to_h_tigs_2.py", line 256, in generate_haplotigs_for_ctg
    extract_and_write_all_ctg(ctg_id, haplotig_graph, all_haplotig_dict, phase_alias_map, out_dir, allow_multiple_primaries, fp_proto_log)
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/falcon_unzip/mains/graphs_to_h_tigs_2.py", line 959, in extract_and_write_all_ctg
    raise Exception(msg)
Exception: Skipping additional subgraphs of the primary contig: 000519F. The graph has multiple primary components.
Traceback (most recent call last):
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/falcon_unzip/mains/graphs_to_h_tigs_2.py", line 1495, in <module>
    main()
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/falcon_unzip/mains/graphs_to_h_tigs_2.py", line 1491, in main
    args.func(args)
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/falcon_unzip/mains/graphs_to_h_tigs_2.py", line 1253, in cmd_apply
    result = run_generate_haplotigs_for_ctg(exe)
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/falcon_unzip/mains/graphs_to_h_tigs_2.py", line 74, in run_generate_haplotigs_for_ctg
    return generate_haplotigs_for_ctg(ctg_id, allow_multiple_primaries, out_dir, unzip_dir, proto_dir, logger)
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/falcon_unzip/mains/graphs_to_h_tigs_2.py", line 256, in generate_haplotigs_for_ctg
    extract_and_write_all_ctg(ctg_id, haplotig_graph, all_haplotig_dict, phase_alias_map, out_dir, allow_multiple_primaries, fp_proto_log)
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/falcon_unzip/mains/graphs_to_h_tigs_2.py", line 959, in extract_and_write_all_ctg
    raise Exception(msg)
Exception: Skipping additional subgraphs of the primary contig: 000519F. The graph has multiple primary components.
2018-09-30 09:11:29,257 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2018-09-30 09:11:29,258 - root - WARNING - CD: '/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F' -> '/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F'
2018-09-30 09:11:29,258 - root - WARNING - CD: '/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F' -> '/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F'
2018-09-30 09:11:29,259 - root - CRITICAL - Error in /home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/pypeflow/do_task.py", line 267, in <module>
    main()
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/pypeflow/do_task.py", line 259, in main
    run(**vars(parsed_args))
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/pypeflow/do_task.py", line 253, in run
    run_cfg_in_tmpdir(cfg, tmpdir, '.')
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
    run_bash(bash_template, myinputs, myoutputs, parameters)
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
    util.system(cmd)
  File "/home/mskkk/.pyenv/versions/anaconda-2.0.0/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
    raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
+++ pwd
++ echo 'FAILURE. Running top in /public/mskkk/Nematode/Mi/WuhanF4-1/01-assembly/falcon/3-unzip/2-htigs/chunk_000519F (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
++ which python
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ pstree -apl

real	0m21.948s
user	0m19.299s
sys	0m2.758s
+ finish
+ echo 'finish code: 1'

Detection of FLNC reads with Isoseq3

Hello,
We are trying to use isoseq to detect new isoforms, and would like to know about the procedure of the detection of the full-length non-chimeric reads (FLNC). To be a FLNC, one read needs to contain: 1) both 5'primer and 3'primer; 2) PolyA signature. 3) Not chimeric, with >1 transcripts joining together.
"Lima" can be used to detect the first signature, with proper 5p--3p orientation. This step may not explore the chimeric signature. Would the "--require-ploya" in cluster step try to find multiple polyAs in the read and remove these kinds of reads? However, I also found quite a lot of reads (~5k reads) were not correctly classified as FLNC reads, with some containing sequencing error in the middle of polyA (e.g., AAAAAAAAGAAAAAAAAAAAAAAAAAAAAA). But again, I also found some CCS reads were not classified as FLNC, even without any sequencing errors.

For example (the first 10 bases are UMI):
m54124_181029_181314/10027310/ccs 20 * 0 255 * * 0 0 GGTGAGAAAGGTTTTTTTTTTTTTTTTTTTTTTTTTAGGGCAGGCCAAACCCTAGTTTATTTCAGTGTCAGCAACAGCTTAGCCATCAAAAAAATAACTCTACCCAGGCGACAGAAGTCTCTACAGCGAGGCTAAGGGTCAGCCGCCAGGCGGCAAACATCAAGGATGCATGGCCGGCACGCCCGGGTAATAAGTTAGGAAGCGGCAGCCTGATGGTGGTGAGGGCCAGGCTTCACTTCTGGGCCGGCATGAGGTCATCGATTGCCTGACCCTGCTCGAGCCGTATTGCTCCATCTCAATGAGTAGTTTCACTCCGTCCACCACCATCTGCACCAGTTCCACCTCCGAGAAGCCCAGGCGGTCAGCGTTAGAGACATCAAACACCCACCGACAGCAGCGGTGTCCACGCCACCTGTGCCTCGCTTCTGAAGCCGCAGCCGCTTGAGCACCTCCGAGAACTTCTCGTGCTTCCCCAGGTGGGGCAGCTTGTGTGCACACCTGCCCGCAGTCCGGTGCTCAGGTTGGATGGGCAGTGTGAGGATGTAGCCCAGGTGAGGATTCCACATGAACTCATAGTCTTGGACTTGAAGAGAGTTTCGATCTGAGTGAGGCCGGTGCAGAATCGGGGAACACTTCCTTCATGTTGCCCCCCTTCTGCATGGAGATGACTCGCAGGTGGTCCTCCTCTTTAATCCACACCAGGGAAAGTCTTATTGTCATTGTGCCATATGCACGAGCATCCGGCCATTCGCTTGGCCATGCCGGAGGCCAGCAGCAGAGGCGACACAGGCTTATCGAAGAGGAAGTGGTCGTCAATGAGCTGCTGCTGCTCCGCCTCAGTCATGCTCGAGCGCGTAGTACCTGCCAACAGGTCGCCATCTAGGCTGGACGAGCTTCTACTGCCACTTCTCGATGGCGCGGCGCTCCCCGCGGCTGGCAGTGCGGGGGAGACAGAAGCCGCGGATGCTGCGGCCTGTGCGCACTCGCGAGCTCAGCACGTAGTTGGGGTCCAGGTCTCGCCACCCTGCAGGTTGTCTGGGTTGAGGTCGGTCTTGTGCTCATCATGGGCTGGTAGCCGCCGTGCCGCTCCTCAATAATGGGGTCCGAAGAGGTCCTTGAATACGTCGTAACTCTCCTCGTCGCCCGCCACTGCACCCACAGTCATGACTGTATCGGGTGGCCCGGATTGTCTACGCCAGTCTGAATGGCGTCGTCCAAAGTAAAGCCGCCGGCGTGCACTTGGCACGGAGCTCGGCGTACAGCTCGGGGTCAGCACCTTGGCATATGGTTGTTGTGGCTGCTCAGATCAGGGAACTCTCCTCGGCCGGAAGCGCAGCTTCTGCGTATTATGGCTGTTGGAGAAGGGCATGGCGGCGGCAGCGGGCGGATGCCGGACGAAGCAGGGCGATGGATGCCTGTGCTTGCAGCTCCTGGCGCAGCGCAGAAGAAAC 55555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555 RG:Z:e175cfed bx:B:i,25,29 np:i:3 rq:f:0 rs:B:i,5,0,0,0,0,0 sn:B:f,6.00709,11.2122,5.56258,9.58404 za:f:0 zm:i:10027310 zs:B:f,0 qs:i:25 qe:i:1476 bc:B:S,1,0 bq:i:95 cx:i:12

Does anyone know how to fix this issue?
Thanks!

Need a mechanism to better control what filesystems are used for running certain steps.

Our Linux cluster has two types of storage, Isilon and Lustre. We have found from running a previous version of Falcon that certain steps require Lustre and other require the Isilon. In particular, LA4Falcon is not happy running on the Isilon -- the seeks to find specific reads can take 100ms on average, whereas our Lustre based filesystem has much better seek performance (single ms or less).

Handling this issue generally is probably too much, but would it be possible to specify an alternative location for running LA4Falcon and having pb-assembly copy the needed files to that location? It's also important that the copy operation not try to copy all the .las files in parallel -- our Lustre filesystem can handle only about 20 simultaneous copying processes at once.

CONDA Solving environment: failed

I tried to install pb-assembly on linux workstation, but it fail to load the environment with pb-assembly ! How to resolve this ?

Conda version: conda 4.5.11

➜  JIT git:(master) ✗ conda update conda                  
Solving environment: done

# All requested packages already installed.

➜  JIT git:(master) ✗ conda install -c bioconda pb-assembly

Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - idna_ssl
  - pb-assembly
Use "conda info <package>" to see the dependencies for each package.

➜ JIT git:(master) ✗ conda list
# packages in environment at /home/urbe/anaconda3:
#
# Name                    Version                   Build  Channel
_ipyw_jlab_nb_ext_conf    0.1.0                    py36_0  
absl-py                   0.4.1                      py_0    conda-forge
afplot                    0.2.1                     <pip>
aioeasywebdav             2.2.0                    py36_0    conda-forge
aiohttp                   3.4.2            py36h470a237_0    conda-forge
alabaster                 0.7.11                     py_3    conda-forge
anaconda                  custom           py36hbbc8b67_0  
anaconda-client           1.7.1                      py_0    conda-forge
anaconda-navigator        1.8.7                    py36_0  
anaconda-project          0.8.2                      py_1    conda-forge
appdirs                   1.4.3                      py_1    conda-forge
argparse                  1.4.0                     <pip>
asn1crypto                0.24.0                   py36_3    conda-forge
astor                     0.7.1                      py_0    conda-forge
astroid                   2.0.2                    py36_0    conda-forge
astropy                   3.0.4                    py36_0    conda-forge
async-timeout             3.0.0                    py36_0    conda-forge
atomicwrites              1.1.5                    py36_0    conda-forge
attrs                     18.1.0                     py_1    conda-forge
automat                   0.7.0                      py_1    conda-forge
babel                     2.6.0                      py_1    conda-forge
backcall                  0.1.0                      py_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.shutil_get_terminal_size 1.0.0                      py_3    conda-forge
bcftools                  1.7                           0    bioconda
bcrypt                    3.1.4            py36h14c3975_0  
beautifulsoup4            4.6.3                    py36_0    conda-forge
bedtools                  2.27.1               he941832_2    bioconda
biopython                 1.72                     py36_0    conda-forge
bitarray                  0.8.3            py36h470a237_0    conda-forge
bkcharts                  0.2                      py36_0    conda-forge
blas                      1.0                         mkl  
blasr                     5.3.2                hac9d22c_3    bioconda
blasr_libcpp              5.3.1                hac9d22c_2    bioconda
blaze                     0.11.3                   py36_0    conda-forge
bleach                    1.4.2                    py36_0    bioconda
blinker                   1.4                        py_1    conda-forge
blist                     1.3.6                     <pip>
blosc                     1.14.3               hdbcaa40_0  
bokeh                     0.13.0                   py36_0    conda-forge
boost                     1.64.0                   py36_4    conda-forge
boost-cpp                 1.64.0                        1    conda-forge
boto                      2.49.0                   py36_0  
boto3                     1.7.80                     py_0    conda-forge
botocore                  1.10.81                    py_0    conda-forge
bottleneck                1.2.1            py36h7eb728f_1    conda-forge
bowtie2                   2.3.4.2          py36h2d50403_0    bioconda
bwa                       0.7.17               ha92aebf_3    bioconda
bz2file                   0.98                     py36_1  
bzip2                     1.0.6                h14c3975_5  
c-ares                    1.14.0               h470a237_0    conda-forge
ca-certificates           2018.8.24            ha4d7672_0    conda-forge
cachetools                2.1.0                      py_0    conda-forge
cairo                     1.14.12              h276e583_5    conda-forge
CAMSA                     1.0.0                     <pip>
certifi                   2018.8.24             py36_1001    conda-forge
cffi                      1.11.5           py36h5e8e0c9_1    conda-forge
chardet                   3.0.4                    py36_3    conda-forge
click                     6.7                        py_1    conda-forge
cloudpickle               0.5.5                      py_0    conda-forge
clyent                    1.2.2                      py_1    conda-forge
colorama                  0.3.9                      py_1    conda-forge
conda                     4.5.11                   py36_0    conda-forge
conda-build               3.10.6                   py36_0    conda-forge
conda-env                 2.6.0                h36134e3_1  
conda-verify              2.0.0                    py36_0    conda-forge
ConfigArgParse            0.13.0                    <pip>
configargparse            0.13.0                     py_1    conda-forge
constantly                15.1.0                     py_0    conda-forge
contextlib2               0.5.5                      py_2    conda-forge
cryptography              2.3.1            py36hdffb7b8_0    conda-forge
cryptography-vectors      2.3.1                    py36_0    conda-forge
curl                      7.60.0               h84994c4_0  
cycler                    0.10.0                     py_1    conda-forge
cython                    0.28.5           py36hfc679d8_0    conda-forge
cytoolz                   0.9.0.1          py36h470a237_0    conda-forge
dask                      0.19.0                     py_0    conda-forge
dask-core                 0.19.0                     py_0    conda-forge
datashape                 0.5.4                    py36_0    conda-forge
datrie                    0.7.1                    py36_0  
dbus                      1.13.2               h714fa37_1  
decorator                 4.3.0                      py_0    conda-forge
deeptools                 3.1.2            py36h470a237_0    bioconda
deepTools-3.1.0-2         38cfe39                   <pip>
dinopy                    2.0.1            py36h470a237_0    bioconda
distributed               1.23.0                   py36_0    conda-forge
docutils                  0.14                     py36_1    conda-forge
dropbox                   9.0.0                      py_0    conda-forge
entrypoints               0.2.3                    py36_2    conda-forge
enum34                    1.1.6                     <pip>
et_xmlfile                1.0.1                    py36_0    conda-forge
expat                     2.2.5                he0dffb1_0  
fastcache                 1.0.2            py36h470a237_1    conda-forge
fastqe                    0.1.1                     <pip>
filechunkio               1.6                      py36_0    bioconda
filelock                  3.0.4                      py_1    conda-forge
flask                     1.0.2                      py_1    conda-forge
flask-cors                3.0.6                      py_0    conda-forge
fontconfig                2.13.1               h65d0f4c_0    conda-forge
freetype                  2.9.1                h6debe1e_4    conda-forge
ftputil                   3.2                      py36_0    bioconda
gast                      0.2.0                      py_0    conda-forge
gensim                    3.5.0                    py36_0    conda-forge
get_terminal_size         1.0.0                haa9412d_0  
gettext                   0.19.8.1             h5e8e0c9_1    conda-forge
gevent                    1.1rc4                   py36_0    bioconda
glib                      2.56.2               h464dc38_0    conda-forge
glob2                     0.4.1                    py36_0    bioconda
gmp                       6.1.2                h6c8ec71_1  
gmpy2                     2.0.8                    py36_1    conda-forge
goleft                    0.1.18                        1    bioconda
google-api-core           0.1.4                      py_0    conda-forge
google-auth               1.2.1                      py_0    conda-forge
google-cloud-core         0.28.1                     py_0    conda-forge
google-cloud-storage      1.10.0                     py_0    conda-forge
google-resumable-media    0.3.1                      py_0    conda-forge
googleapis-common-protos  1.5.3                      py_1    conda-forge
graphite2                 1.3.11               h16798f4_2  
graphviz                  2.38.0               h08bfae6_9    conda-forge
greenlet                  0.4.13                   py36_0    conda-forge
grpcio                    1.14.1           py36hd60e7a3_0    conda-forge
gsl                       2.2.1                h0c605f7_3  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
h5py                      2.8.0            py36hb794570_1    conda-forge
harfbuzz                  1.9.0                hee26f79_1    conda-forge
hdf5                      1.10.2               hba1933b_1  
heapdict                  1.0.0                    py36_0    conda-forge
html5lib                  1.0.1                      py_0    conda-forge
htslib                    1.7                           0    bioconda
hyperlink                 17.3.1                     py_0    conda-forge
icu                       58.2                 h9c2bf20_1  
idna                      2.7                      py36_2    conda-forge
idna_ssl                  1.0.0                         0    conda-forge
imageio                   2.3.0                      py_1    conda-forge
imagesize                 1.0.0                      py_1    conda-forge
incremental               17.5.0                     py_0    conda-forge
intel-openmp              2018.0.0                      8  
ipykernel                 4.9.0                    py36_0    conda-forge
ipython                   6.5.0                    py36_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.4.0                      py_0    conda-forge
isort                     4.3.4                    py36_0    conda-forge
itsdangerous              0.24                       py_2    conda-forge
jbig                      2.1                  hdba287a_0  
jdcal                     1.4                        py_1    conda-forge
jedi                      0.12.1                   py36_0    conda-forge
jeepney                   0.3.1                      py_0    conda-forge
jemalloc                  5.1.0                hfc679d8_0    conda-forge
jinja2                    2.10                       py_1    conda-forge
jmespath                  0.9.3                      py_1    conda-forge
jpeg                      9c                   h470a237_1    conda-forge
jsonschema                2.6.0                    py36_2    conda-forge
jupyter                   1.0.0                      py_1    conda-forge
jupyter_client            5.2.3                      py_1    conda-forge
jupyter_console           5.2.0                    py36_1    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
jupyterlab                0.34.7                   py36_0    conda-forge
jupyterlab_launcher       0.13.1                     py_2    conda-forge
kat                       2.4.1            py36h355e19c_2    bioconda
keras                     2.2.2                    py36_0    conda-forge
keras-applications        1.0.4                      py_1    conda-forge
keras-preprocessing       1.0.2                      py_1    conda-forge
keyring                   13.2.1                   py36_0    conda-forge
kiwisolver                1.0.1            py36h2d50403_2    conda-forge
krb5                      1.14.6                        0    conda-forge
lazy-object-proxy         1.3.1            py36h470a237_0    conda-forge
libcurl                   7.60.0               h1ad7b7a_0  
libedit                   3.1.20170329         h6b74fdf_2  
libffi                    3.2.1                hd88cf55_4  
libgcc                    7.2.0                h69d50b8_2  
libgcc-ng                 7.2.0                hdf63c60_3  
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3  
libgpuarray               0.7.5                         0    conda-forge
libiconv                  1.15                 h470a237_3    conda-forge
libpng                    1.6.34               hb9fc6fc_0  
libprotobuf               3.6.0                hd28b015_0    conda-forge
libsodium                 1.0.16               h1bed415_0  
libssh2                   1.8.0                h9cfc8f7_4  
libstdcxx-ng              7.2.0                hdf63c60_3  
libtiff                   4.0.9                he85c1e1_1  
libtool                   2.4.6                h544aabb_3  
libuuid                   2.32.1               h470a237_2    conda-forge
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.8                h26e45fe_1  
libxslt                   1.1.32               h1312cb7_0  
llvmlite                  0.23.0                   py36_1    conda-forge
locket                    0.2.0                      py_2    conda-forge
lordec                    0.9                           0    atgc-montpellier
lxml                      4.2.4            py36hc9114bc_0    conda-forge
lzo                       2.10                 h49e0be7_2  
mako                      1.0.7                      py_1    conda-forge
markdown                  2.6.11                     py_0    conda-forge
markupsafe                1.0              py36h470a237_1    conda-forge
matplotlib                2.2.3            py36hb69df0a_0  
mccabe                    0.6.1                      py_1    conda-forge
mistune                   0.8.3            py36h470a237_2    conda-forge
mkl                       2018.0.2                      1  
mkl-service               1.1.2            py36h651fb7a_4  
mkl_fft                   1.0.6                    py36_0    conda-forge
mkl_random                1.0.1                    py36_0    conda-forge
more-itertools            4.3.0                    py36_0    conda-forge
mpc                       1.0.3                hec55b23_5  
mpfr                      3.1.5                h11a74b3_2  
mpmath                    1.0.0                      py_0    conda-forge
msgpack-python            0.5.6            py36h2d50403_2    conda-forge
multidict                 4.3.1            py36h470a237_0    conda-forge
multipledispatch          0.6.0                      py_0    conda-forge
mummer                    3.23                    pl526_6    bioconda
mummer4                   4.0.0beta2      pl526hfc679d8_2    bioconda
natsort                   5.3.3                     <pip>
navigator-updater         0.2.1                    py36_0  
nbconvert                 5.3.1                      py_1    conda-forge
nbformat                  4.4.0                      py_1    conda-forge
ncurses                   6.1                  hf484d3e_0  
networkx                  2.1                        py_1    conda-forge
nltk                      3.2.5                      py_0    conda-forge
nose                      1.3.7                    py36_2    conda-forge
notebook                  5.6.0                    py36_1    conda-forge
numba                     0.38.1                   py36_0    conda-forge
numexpr                   2.6.6                    py36_0    conda-forge
numpy                     1.14.3           py36hcd700cb_1  
numpy-base                1.14.3           py36h9be14a7_1  
numpydoc                  0.8.0                      py_1    conda-forge
oauthlib                  2.1.0                      py_0    conda-forge
odo                       0.5.1                      py_1    conda-forge
olefile                   0.45.1                     py_1    conda-forge
openjdk                   8.0.121                       1  
openpyxl                  2.4.0                    py36_0    bioconda
openssl                   1.0.2p               h470a237_0    conda-forge
packaging                 17.1                       py_0    conda-forge
pandas                    0.23.4           py36hf8a1672_0    conda-forge
pandoc                    1.19.2.1             hea2e7c5_1  
pandocfilters             1.4.2                      py_1    conda-forge
pango                     1.40.14              he752989_2    conda-forge
paramiko                  2.1.2                    py36_0    conda-forge
parso                     0.3.1                      py_0    conda-forge
partd                     0.3.8                      py_1    conda-forge
patchelf                  0.9                  hf79760b_2  
path.py                   11.0.1                     py_0    conda-forge
pathlib2                  2.3.2                    py36_0    conda-forge
patsy                     0.5.0                      py_1    conda-forge
pbbam                     0.18.0               h1310cd9_1    bioconda
pcre                      8.42                 h439df22_0  
pep8                      1.7.1                      py_0    conda-forge
perl                      5.26.2               h470a237_0    conda-forge
perl-app-cpanminus        1.7044                  pl526_1    bioconda
perl-carp                 1.38                    pl526_1    bioconda
perl-constant             1.33                    pl526_1    bioconda
perl-cpan-meta            2.120921                pl526_1    bioconda
perl-cpan-shell           5.5004                  pl526_1    bioconda
perl-data-dumper          2.161                   pl526_2    bioconda
perl-env-path             0.19                          0    bioconda
perl-exporter             5.72                    pl526_1    bioconda
perl-extutils-cbuilder    0.280230                pl526_1    bioconda
perl-extutils-makemaker   7.34                    pl526_1    bioconda
perl-extutils-manifest    1.70                    pl526_2    bioconda
perl-extutils-parsexs     3.35                    pl526_0    bioconda
perl-file-path            2.15                    pl526_0    bioconda
perl-file-temp            0.2304                  pl526_2    bioconda
perl-getopt-long          2.50                    pl526_1    bioconda
perl-ipc-cmd              1.02                    pl526_0    bioconda
perl-list-util            1.38                    pl526_1    bioconda
perl-locale-maketext-simple 0.21                    pl526_1    bioconda
perl-module-build         0.4224                  pl526_3    bioconda
perl-module-corelist      5.20180626              pl526_0    bioconda
perl-module-load          0.32                    pl526_1    bioconda
perl-module-load-conditional 0.68                    pl526_2    bioconda
perl-module-metadata      1.000033                pl526_0    bioconda
perl-params-check         0.38                    pl526_1    bioconda
perl-parent               0.236                   pl526_1    bioconda
perl-perl-ostype          1.010                   pl526_1    bioconda
perl-scalar-list-utils    1.45            pl526h470a237_3    bioconda
perl-test-more            1.001002                pl526_1    bioconda
perl-text-abbrev          1.02                    pl526_0    bioconda
perl-text-parsewords      3.29                    pl526_3    bioconda
perl-threaded             5.22.0                       13    bioconda
perl-version              0.9924                  pl526_0    bioconda
pexpect                   4.6.0                    py36_0    conda-forge
pickleshare               0.7.4                    py36_0    conda-forge
pillow                    5.2.0            py36hc736899_1    conda-forge
pilon                     1.22                          1    bioconda
pip                       18.0                      <pip>
pip                       10.0.1                    <pip>
pip                       18.0                     py36_1    conda-forge
pixman                    0.34.0               hceecf20_3  
pkginfo                   1.4.2                      py_1    conda-forge
plotly                    2.7.0                      py_1    conda-forge
pluggy                    0.7.1                      py_0    conda-forge
ply                       3.11                       py_1    conda-forge
porechop                  0.2.3                     <pip>
prettytable               0.7.2                      py_2    conda-forge
progressbar2              3.38.0                    <pip>
prometheus_client         0.3.0                      py_0    conda-forge
prompt_toolkit            1.0.15                     py_1    conda-forge
protobuf                  3.6.0            py36hfc679d8_0    conda-forge
psutil                    5.4.7            py36h470a237_1    conda-forge
ptyprocess                0.6.0                    py36_0    conda-forge
py                        1.6.0                      py_0    conda-forge
py2bit                    0.3.0                    py36_1    bioconda
pyasn1                    0.4.4                      py_0    conda-forge
pyasn1-modules            0.0.5                    py36_0    bioconda
pybedtools                0.7.10                    <pip>
pybigwig                  0.3.11           py36he7a497f_2    bioconda
pycodestyle               2.4.0                      py_1    conda-forge
pycosat                   0.6.3            py36h470a237_1    conda-forge
pycparser                 2.18                       py_1    conda-forge
pycrypto                  2.6.1                    py36_1    conda-forge
pycurl                    7.43.0.2         py36hb7f436b_0  
pyemojify                 0.2.0                     <pip>
pyfaidx                   0.5.4.2                   <pip>
pyflakes                  2.0.0                      py_0    conda-forge
pygments                  2.2.0                      py_1    conda-forge
pygpu                     0.7.5                    py36_0    conda-forge
pygraphviz                1.3.1                    py36_0    bioconda
pyhamcrest                1.9.0                      py_2    conda-forge
PyIntervalTree            0.5                       <pip>
pyjwt                     1.6.4                      py_0    conda-forge
pylint                    2.1.1                    py36_0    conda-forge
pynacl                    0.3.0                    py36_0    bioconda
pyodbc                    4.0.23           py36hfc679d8_1    conda-forge
pyopenssl                 18.0.0                   py36_0    conda-forge
pyparsing                 2.2.0                      py_1    conda-forge
pyqt                      5.9.2            py36h751905a_0  
pysam                     0.8.4                     <pip>
pysam                     0.14.1           py36hae42fb6_1    bioconda
pysam                     0.14.1                    <pip>
pysftp                    0.2.9                    py36_0    bioconda
pysocks                   1.6.8                    py36_2    conda-forge
pytables                  3.4.4            py36h4f72b40_1    conda-forge
pytest                    3.7.4                    py36_0    conda-forge
pytest-arraydiff          0.2                        py_0    conda-forge
pytest-astropy            0.4.0                      py_0    conda-forge
pytest-doctestplus        0.1.3                      py_0    conda-forge
pytest-openfiles          0.3.0                      py_0    conda-forge
pytest-remotedata         0.3.0                      py_0    conda-forge
python                    3.6.6                h5001a0f_0    conda-forge
python-crfsuite           0.9.6            py36h470a237_0    conda-forge
python-dateutil           2.7.3                      py_0    conda-forge
python-igraph             0.7.1.post6               <pip>
python-irodsclient        0.7.0                      py_0    conda-forge
python-utils              2.3.0                     <pip>
pytz                      2018.5                     py_0    conda-forge
PyVCF                     0.6.8                     <pip>
pywavelets                1.0.0            py36h7eb728f_0    conda-forge
pyyaml                    3.13             py36h470a237_1    conda-forge
pyzmq                     17.1.2           py36hae99301_0    conda-forge
qt                        5.9.6                h52aff34_0  
qtawesome                 0.4.4              pyh8a2030e_1    conda-forge
qtconsole                 4.4.1                    py36_1    conda-forge
qtpy                      1.5.0              pyh8a2030e_0    conda-forge
r                         3.5.1                    r351_0    conda-forge
r-base                    3.5.1                h4fe35fd_0    conda-forge
r-boot                    1.3_20                   r351_0    conda-forge
r-class                   7.3_14           r351hc070d10_2    conda-forge
r-cluster                 2.0.7_1          r351h364d78e_0    conda-forge
r-codetools               0.2_15           r351h6115d3f_1    conda-forge
r-foreign                 0.8_71           r351hc070d10_2    conda-forge
r-kernsmooth              2.23_15          r351h364d78e_2    conda-forge
r-lattice                 0.20_35          r351hc070d10_0    conda-forge
r-mass                    7.3_50           r351hc070d10_2    conda-forge
r-matrix                  1.2_14           r351hc070d10_2    conda-forge
r-mgcv                    1.8_24           r351hc070d10_2    conda-forge
r-nlme                    3.1_137          r351h364d78e_0    conda-forge
r-nnet                    7.3_12           r351hc070d10_2    conda-forge
r-recommended             3.5.1                    r351_1    conda-forge
r-rpart                   4.1_13           r351hc070d10_2    conda-forge
r-spatial                 7.3_11           r351hc070d10_2    conda-forge
r-survival                2.42_6           r351hc070d10_1    conda-forge
ratelimiter               1.2.0                    py36_0    conda-forge
readline                  7.0                  ha6073c6_4  
requests                  2.19.1                   py36_1    conda-forge
requests-oauthlib         1.0.0                      py_1    conda-forge
rope                      0.10.7                     py_1    conda-forge
rsa                       3.1.4                    py36_0    bioconda
ruamel_yaml               0.15.63          py36h470a237_0    conda-forge
s3transfer                0.1.13                   py36_0    conda-forge
salmon                    0.10.2                        1    bioconda
samtools                  1.7                           1    bioconda
scikit-image              0.14.0           py36hfc679d8_1    conda-forge
scikit-learn              0.19.1           py36h7aa7ec6_0  
scipy                     1.1.0            py36hfc37229_0  
seaborn                   0.9.0                      py_0    conda-forge
secretstorage             3.0.1                    py36_0    conda-forge
send2trash                1.5.0                      py_0    conda-forge
service_identity          17.0.0                     py_0    conda-forge
setuptools                40.2.0                   py36_0    conda-forge
simlord                   1.0.2            py36h24bf2e0_2    bioconda
simplegeneric             0.8.1                      py_1    conda-forge
singledispatch            3.4.0.3                  py36_0    conda-forge
sip                       4.19.8           py36hfc679d8_0    conda-forge
six                       1.11.0                   py36_1    conda-forge
smart_open                1.6.0                      py_1    conda-forge
snakemake-minimal         5.2.2                    py36_1    bioconda
snappy                    1.1.7                hbae5bb6_3  
snowballstemmer           1.2.1                      py_1    conda-forge
sortedcollections         1.0.1                      py_1    conda-forge
sortedcontainers          2.0.4                      py_1    conda-forge
sphinx                    1.7.5                    py36_0    conda-forge
sphinxcontrib             1.0                      py36_1  
sphinxcontrib-websupport  1.1.0                      py_1    conda-forge
spyder                    3.3.1                    py36_1    conda-forge
spyder-kernels            0.2.6                      py_0    conda-forge
sqlalchemy                1.2.11           py36h470a237_0    conda-forge
sqlite                    3.24.0               h2f33b56_0    conda-forge
statsmodels               0.9.0                    py36_0    conda-forge
sympy                     1.2                      py36_0    conda-forge
tabulate                  0.8.2                      py_0    conda-forge
tbb                       2018_20171205                 0    conda-forge
tblib                     1.3.2                      py_1    conda-forge
tensorboard               1.10.0                   py36_0    conda-forge
tensorflow                1.10.0                   py36_0    conda-forge
termcolor                 1.1.0                      py_2    conda-forge
terminado                 0.8.1                    py36_1    conda-forge
testpath                  0.3.1                    py36_1    conda-forge
theano                    1.0.2                    py36_0    conda-forge
tk                        8.6.8                         0    conda-forge
toolz                     0.9.0                      py_0    conda-forge
tornado                   5.1              py36h470a237_1    conda-forge
traitlets                 4.3.2                    py36_0    conda-forge
twisted                   18.7.0           py36h470a237_1    conda-forge
twython                   3.7.0                      py_0    conda-forge
typed-ast                 1.1.0                    py36_0    conda-forge
typing                    3.5.2.2                  py36_0    bioconda
unicodecsv                0.14.1                     py_1    conda-forge
unixodbc                  2.3.6                h1bed415_0  
urllib3                   1.23                     py36_1    conda-forge
vcftools                  0.1.16               he941832_2    bioconda
virtualenv                15.1.0                   py36_0    conda-forge
wcwidth                   0.1.7                      py_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
werkzeug                  0.14.1                     py_0    conda-forge
wheel                     0.31.1                   py36_1    conda-forge
widgetsnbextension        3.4.1                    py36_0    conda-forge
wrapt                     1.10.11                  py36_0    conda-forge
xlrd                      1.1.0                      py_2    conda-forge
xlsxwriter                1.0.9                      py_0    conda-forge
xlwt                      1.3.0                      py_1    conda-forge
xmlrunner                 1.7.7                      py_0    conda-forge
xorg-kbproto              1.0.7                h470a237_2    conda-forge
xorg-libice               1.0.9                h470a237_4    conda-forge
xorg-libsm                1.2.2                h8c8a85c_6    conda-forge
xorg-libx11               1.6.6                h470a237_0    conda-forge
xorg-libxdmcp             1.1.2                h470a237_7    conda-forge
xorg-libxext              1.3.3                h470a237_4    conda-forge
xorg-libxpm               3.5.12               h470a237_2    conda-forge
xorg-libxrender           0.9.10               h470a237_2    conda-forge
xorg-libxt                1.1.5                h470a237_2    conda-forge
xorg-renderproto          0.11.1               h470a237_2    conda-forge
xorg-xextproto            7.3.0                h470a237_2    conda-forge
xorg-xproto               7.0.31               h470a237_7    conda-forge
xz                        5.2.4                h14c3975_4  
yaml                      0.1.7                had09818_2  
yarl                      1.2.6            py36h470a237_0    conda-forge
zeromq                    4.2.5                h439df22_0  
zict                      0.1.3                      py_0    conda-forge
zlib                      1.2.11               ha838bed_2  
zope.interface            4.5.0            py36h470a237_1    conda-forge

pacificbiosciences / pbbioconda Goto Github PK

pbbioconda's Introduction

Information

Support

File sharing

Installation

Availability

Packages

Combo-Packages

Available Meta-Packages

FAQ

Where do I get support?

I can't find tool X, when will it be available on bioconda?

When will a new version of tool X be available?

Will SMRT Link be available on bioconda?

Can I get the exact version of the binaries from SMRT Link version XY through bioconda?

Which version of tool X shall I use, from bioconda or from SMRT Link?

Why is the official SMRT Tools Guide not in sync with bioconda binaries?

Which operating systems are supported?

Disclaimer/Copyright

pbbioconda's People

Contributors

Stargazers

Watchers

Forkers

pbbioconda's Issues

Name Version Build Channel

To reproduce:

Create environment:

Activate environment

Run bax2bam

Further info:

Even further info:

Recommend Projects

Recommend Topics

Recommend Org

Jobs