[ERROR]Task Node(0-rawreads/build) failed with exit-code=1 about pbbioconda

[gbdias]

• It worked this time!
  Thanks a lot guys

Guilherme

[gbdias]

• Here is the all.log file.

2018-09-25 18:43:10,617 - fc_run:537 - INFO - Setup logging from file "None".
2018-09-25 18:43:10,624 - root:43 - INFO - $ lfs setstripe -c 12 /lustre1/gd98309/funzip/ecoli3 >
2018-09-25 18:43:10,636 - root:57 - WARNING - 'lfs setstripe -c 12 /lustre1/gd98309/funzip/ecoli3' failed to produ
ce any output.
2018-09-25 18:43:10,637 - falcon_kit.util.system:51 - INFO - Lustre filesystem detected. This lfs stripe (12) shou
ld propagate to subdirs of '/lustre1/gd98309/funzip/ecoli3'.
2018-09-25 18:43:10,637 - fc_run:50 - INFO - fc_run started with configuration fc_run.cfg
2018-09-25 18:43:10,787 - fc_run:475 - WARNING - Unexpected keys in input config: set(['ovlp_HPCTANmask_option'])
2018-09-25 18:43:10,788 - fc_run:55 - INFO - cfg=
{
  "General": {
    "LA4Falcon_preload": false,
    "avoid_text_file_busy": true,
    "bestn": 12,
    "dazcon": false,
    "falcon_sense_greedy": false,
    "falcon_sense_option": "--output-multi --min-idt 0.70 --min-cov 2 --max-n-read 400",
    "falcon_sense_skip_contained": false,
    "fc_ovlp_to_graph_option": " --min-len 1000",
    "genome_size": "4600000",
    "input_fofn": "input.fofn",
    "input_type": "raw",
    "length_cutoff": "-1",
    "length_cutoff_pr": "1000",
    "overlap_filtering_setting": "--max-diff 100 --max-cov 100 --min-cov 2",
    "ovlp_DBsplit_option": "-x500 -s200",
    "ovlp_HPCTANmask_option": "-l500",
    "ovlp_HPCdaligner_option": "-v -B128 -M24",
    "ovlp_daligner_option": "-e.93 -l2500 -k24 -h1024 -w6 -s100",
    "pa_DBdust_option": "True",
    "pa_DBsplit_option": "-x500 -s200",
    "pa_HPCTANmask_option": "",
    "pa_HPCdaligner_option": "-v -B128 -M24",
    "pa_REPmask_code": "0,300;0,300;0,300",
    "pa_daligner_option": "-e.8 -l2000 -k18 -h480  -w8 -s100",
    "pa_dazcon_option": "-j 4 -x -l 500",
    "pa_fasta_filter_option": "pass",
    "seed_coverage": "20",
    "skip_checks": true,
    "target": "assembly"
  },
  "job.defaults": {
    "JOB_QUEUE": "default",
    "MB": "32768",
    "NPROC": "8",
    "job_type": "local",
    "njobs": "32",
    "pwatcher_type": "blocking",
    "submit": "qsub -S /bin/bash -sync y -V  \\\n-q ${JOB_QUEUE}     \\\n-N ${JOB_NAME}      \\\n-o \"${JOB_STDOUT
}\"  \\\n-e \"${JOB_STDERR}\"  \\\n-pe smp ${NPROC}    \\\n-l h_vmem=${MB}M    \\\n\"${JOB_SCRIPT}\"",
    "use_tmpdir": false
  },
  "job.step.asm": {
    "MB": "32768",
    "NPROC": "4",
    "njobs": "3"
  },
  "job.step.cns": {
    "MB": "49152",
    "NPROC": "4",
    "njobs": "2"
  },
  "job.step.da": {},
  "job.step.la": {},
  "job.step.pda": {},
  "job.step.pla": {}
}
2018-09-25 18:43:10,788 - pypeflow.simple_pwatcher_bridge:723 - INFO - In simple_pwatcher_bridge, pwatcher_impl=<m
odule 'pwatcher.blocking' from '/home/gd98309/.conda/envs/falconda/lib/python2.7/site-packages/pwatcher/blocking.p
yc'>
2018-09-25 18:43:10,788 - pypeflow.simple_pwatcher_bridge:734 - INFO - job_type='local', (default)job_defaults={'J
OB_QUEUE': 'default', 'pwatcher_type': 'blocking', 'use_tmpdir': False, 'MB': '32768', 'job_type': 'local', 'submi
t': 'qsub -S /bin/bash -sync y -V  \\\n-q ${JOB_QUEUE}     \\\n-N ${JOB_NAME}      \\\n-o "${JOB_STDOUT}"  \\\n-e 
"${JOB_STDERR}"  \\\n-pe smp ${NPROC}    \\\n-l h_vmem=${MB}M    \\\n"${JOB_SCRIPT}"', 'NPROC': '8', 'njobs': '32'
}, use_tmpdir=False, squash=False, job_name_style=0
2018-09-25 18:43:10,789 - pypeflow.simple_pwatcher_bridge:376 - INFO - Setting max_jobs to 32; was None
2018-09-25 18:43:10,789 - root:23 - DEBUG - 'Serializing 8 records'
2018-09-25 18:43:10,987 - root:23 - DEBUG - "  Opened for write: './config.json'"
2018-09-25 18:43:10,988 - root:23 - DEBUG - '  Serialized to 0.0MB as json'
2018-09-25 18:43:10,989 - root:23 - DEBUG - "Deserializing from '/lustre1/gd98309/funzip/ecoli3/config.json'"
2018-09-25 18:43:10,989 - root:23 - DEBUG - "  Opened for read: '/lustre1/gd98309/funzip/ecoli3/config.json'"
2018-09-25 18:43:10,990 - root:23 - DEBUG - '  Read 0.0MB as json'
2018-09-25 18:43:10,990 - root:23 - DEBUG - '  Deserialized 8 records'
2018-09-25 18:43:10,990 - root:23 - DEBUG - 'Serializing 30 records'
2018-09-25 18:43:11,009 - root:23 - DEBUG - "  Opened for write: '/lustre1/gd98309/funzip/ecoli3/General_config.js
on'"
2018-09-25 18:43:11,010 - root:23 - DEBUG - '  Serialized to 0.0MB as json'
2018-09-25 18:43:11,011 - falcon_kit.util.system:97 - DEBUG - mkdir -p '0-rawreads'
2018-09-25 18:43:11,011 - falcon_kit.util.system:97 - DEBUG - mkdir -p '1-preads_ovl'
2018-09-25 18:43:11,012 - falcon_kit.util.system:97 - DEBUG - mkdir -p '2-asm-falcon'
2018-09-25 18:43:11,012 - pypeflow.tasks:45 - DEBUG - gen_task(python -m falcon_kit.mains.dazzler --config-fn={inp
ut.config} --db-fn={output.db}  build --input-fofn-fn={input.input_fofn} --length-cutoff-fn={output.length_cutoff}
# TODO: Verify that db exists.
#ln -sf {output.length_cutoff} length_cutoff

	inputs={'config': 'General_config.json', 'input_fofn': 'input.fofn'},
	outputs={'length_cutoff': '0-rawreads/build/length_cutoff', 'db': '0-rawreads/build/raw_reads.db'})
2018-09-25 18:43:11,013 - pypeflow.tasks:50 - DEBUG -  parameters={'pypeflow_mb': 4000, 'pypeflow_nproc': 1, 'topd
ir': '../..'}
2018-09-25 18:43:11,013 - pypeflow.tasks:52 - DEBUG -  dist.job_dict={'MB': 4000, 'NPROC': 1}
2018-09-25 18:43:11,015 - pypeflow.simple_pwatcher_bridge:619 - DEBUG - New PypeTask(wdir=None,
	inputs={'input_fofn': 'input.fofn', 'config': 'General_config.json'},
	outputs={'length_cutoff': '0-rawreads/build/length_cutoff', 'db': '0-rawreads/build/raw_reads.db'})
2018-09-25 18:43:11,016 - pypeflow.simple_pwatcher_bridge:698 - DEBUG - Created PypeTask('0-rawreads/build', '/lus
tre1/gd98309/funzip/ecoli3/0-rawreads/build', "{'db': '/lustre1/gd98309/funzip/ecoli3/0-rawreads/build/raw_reads.d
b',\n 'length_cutoff': '/lustre1/gd98309/funzip/ecoli3/0-rawreads/build/length_cutoff'}", "{'config': '/lustre1/gd
98309/funzip/ecoli3/General_config.json',\n 'input_fofn': '/lustre1/gd98309/funzip/ecoli3/input.fofn'}")
2018-09-25 18:43:11,016 - pypeflow.simple_pwatcher_bridge:641 - DEBUG - Added PRODUCERS['0-rawreads/build'] = Pype
Task('0-rawreads/build', '/lustre1/gd98309/funzip/ecoli3/0-rawreads/build', "{'db': '/lustre1/gd98309/funzip/ecoli
3/0-rawreads/build/raw_reads.db',\n 'length_cutoff': '/lustre1/gd98309/funzip/ecoli3/0-rawreads/build/length_cutof
f'}", "{'config': '/lustre1/gd98309/funzip/ecoli3/General_config.json',\n 'input_fofn': '/lustre1/gd98309/funzip/e
coli3/input.fofn'}")
2018-09-25 18:43:11,017 - pypeflow.simple_pwatcher_bridge:547 - DEBUG - No producer PypeTask for basedir '.' from 
path '/lustre1/gd98309/funzip/ecoli3/input.fofn' -- Pure input?
2018-09-25 18:43:11,017 - pypeflow.simple_pwatcher_bridge:547 - DEBUG - No producer PypeTask for basedir '.' from 
path '/lustre1/gd98309/funzip/ecoli3/General_config.json' -- Pure input?
2018-09-25 18:43:11,018 - pypeflow.simple_pwatcher_bridge:645 - DEBUG - Built PypeTask('0-rawreads/build', '/lustr
e1/gd98309/funzip/ecoli3/0-rawreads/build', "{'db': PLF('raw_reads.db', '0-rawreads/build'),\n 'length_cutoff': PL
F('length_cutoff', '0-rawreads/build')}", "{'config': PLF('General_config.json', None),\n 'input_fofn': PLF('input
.fofn', None)}")
2018-09-25 18:43:11,018 - pypeflow.simple_pwatcher_bridge:259 - DEBUG - New Node(0-rawreads/build) needs set([])
2018-09-25 18:43:11,018 - pypeflow.tasks:45 - DEBUG - gen_task(python -m falcon_kit.mains.dazzler --config={input.
config} --db={input.db}  tan-split --split={output.split} --bash-template={output.bash_template}

	inputs={'config': 'General_config.json', 'db': '0-rawreads/build/raw_reads.db'},
	outputs={'split': '0-rawreads/tan-split/tan-uows.json', 'bash_template': '0-rawreads/tan-split/bash_templa
te.sh'})
2018-09-25 18:43:11,018 - pypeflow.tasks:50 - DEBUG -  parameters={'pypeflow_mb': 4000, 'pypeflow_nproc': 1, 'topd
ir': '../..'}
2018-09-25 18:43:11,019 - pypeflow.tasks:52 - DEBUG -  dist.job_dict={'MB': 4000, 'NPROC': 1}
2018-09-25 18:43:11,020 - pypeflow.simple_pwatcher_bridge:619 - DEBUG - New PypeTask(wdir=None,
	inputs={'db': '0-rawreads/build/raw_reads.db', 'config': 'General_config.json'},
	outputs={'split': '0-rawreads/tan-split/tan-uows.json', 'bash_template': '0-rawreads/tan-split/bash_templa
te.sh'})
2018-09-25 18:43:11,021 - pypeflow.simple_pwatcher_bridge:698 - DEBUG - Created PypeTask('0-rawreads/tan-split', '
/lustre1/gd98309/funzip/ecoli3/0-rawreads/tan-split', "{'bash_template': '/lustre1/gd98309/funzip/ecoli3/0-rawread
s/tan-split/bash_template.sh',\n 'split': '/lustre1/gd98309/funzip/ecoli3/0-rawreads/tan-split/tan-uows.json'}", "
{'config': '/lustre1/gd98309/funzip/ecoli3/General_config.json',\n 'db': '/lustre1/gd98309/funzip/ecoli3/0-rawread
s/build/raw_reads.db'}")
2018-09-25 18:43:11,022 - pypeflow.simple_pwatcher_bridge:641 - DEBUG - Added PRODUCERS['0-rawreads/tan-split'] = 
PypeTask('0-rawreads/tan-split', '/lustre1/gd98309/funzip/ecoli3/0-rawreads/tan-split', "{'bash_template': '/lustr
e1/gd98309/funzip/ecoli3/0-rawreads/tan-split/bash_template.sh',\n 'split': '/lustre1/gd98309/funzip/ecoli3/0-rawr
eads/tan-split/tan-uows.json'}", "{'config': '/lustre1/gd98309/funzip/ecoli3/General_config.json',\n 'db': '/lustr
e1/gd98309/funzip/ecoli3/0-rawreads/build/raw_reads.db'}")
2018-09-25 18:43:11,022 - pypeflow.simple_pwatcher_bridge:547 - DEBUG - No producer PypeTask for basedir '.' from 
path '/lustre1/gd98309/funzip/ecoli3/General_config.json' -- Pure input?
2018-09-25 18:43:11,023 - pypeflow.simple_pwatcher_bridge:645 - DEBUG - Built PypeTask('0-rawreads/tan-split', '/l
ustre1/gd98309/funzip/ecoli3/0-rawreads/tan-split', "{'bash_template': PLF('bash_template.sh', '0-rawreads/tan-spl
it'),\n 'split': PLF('tan-uows.json', '0-rawreads/tan-split')}", "{'config': PLF('General_config.json', None),\n '
db': PLF('raw_reads.db', '0-rawreads/build')}")
2018-09-25 18:43:11,023 - pypeflow.simple_pwatcher_bridge:259 - DEBUG - New Node(0-rawreads/tan-split) needs set([
Node(0-rawreads/build)])
2018-09-25 18:43:11,025 - pypeflow.simple_pwatcher_bridge:304 - INFO - Num unsatisfied: 2, graph: 2
2018-09-25 18:43:11,025 - pypeflow.simple_pwatcher_bridge:314 - INFO - About to submit: Node(0-rawreads/build)
2018-09-25 18:43:11,025 - pypeflow.simple_pwatcher_bridge:109 - DEBUG - enque nodes:
set([Node(0-rawreads/build)])
2018-09-25 18:43:11,026 - root:17 - DEBUG - mkdir -p "/lustre1/gd98309/funzip/ecoli3/0-rawreads/build"
2018-09-25 18:43:12,405 - pypeflow.simple_pwatcher_bridge:132 - DEBUG - In rundir='/lustre1/gd98309/funzip/ecoli3/
0-rawreads/build', job_dict={'JOB_QUEUE': 'default', 'pwatcher_type': 'blocking', 'use_tmpdir': False, 'MB': 4000,
 'job_type': 'local', 'submit': 'qsub -S /bin/bash -sync y -V  \\\n-q ${JOB_QUEUE}     \\\n-N ${JOB_NAME}      \\\
n-o "${JOB_STDOUT}"  \\\n-e "${JOB_STDERR}"  \\\n-pe smp ${NPROC}    \\\n-l h_vmem=${MB}M    \\\n"${JOB_SCRIPT}"',
 'NPROC': 1, 'njobs': '32'}
2018-09-25 18:43:12,406 - pwatcher.blocking:459 - DEBUG - run(jobids=<1>, job_type=local, job_defaults_dict={'JOB_
QUEUE': 'default', 'pwatcher_type': 'blocking', 'use_tmpdir': False, 'MB': '32768', 'job_type': 'local', 'submit':
 'qsub -S /bin/bash -sync y -V  \\\n-q ${JOB_QUEUE}     \\\n-N ${JOB_NAME}      \\\n-o "${JOB_STDOUT}"  \\\n-e "${
JOB_STDERR}"  \\\n-pe smp ${NPROC}    \\\n-l h_vmem=${MB}M    \\\n"${JOB_SCRIPT}"', 'NPROC': '8', 'njobs': '32'})
2018-09-25 18:43:12,406 - pwatcher.blocking:348 - DEBUG - NOTE: In blocking pwatcher, job_type='local', should be 
'string'
2018-09-25 18:43:12,406 - pwatcher.blocking:363 - DEBUG - jobs:
{'Pa93d51772c1627': Job(jobid='Pa93d51772c1627', cmd='/bin/bash run.sh', rundir='/lustre1/gd98309/funzip/ecoli3/0-
rawreads/build', options={'JOB_QUEUE': 'default', 'pwatcher_type': 'blocking', 'use_tmpdir': False, 'MB': 4000, 'j
ob_type': 'local', 'submit': 'qsub -S /bin/bash -sync y -V  \\\n-q ${JOB_QUEUE}     \\\n-N ${JOB_NAME}      \\\n-o
 "${JOB_STDOUT}"  \\\n-e "${JOB_STDERR}"  \\\n-pe smp ${NPROC}    \\\n-l h_vmem=${MB}M    \\\n"${JOB_SCRIPT}"', 'N
PROC': 1, 'njobs': '32', 'local': 0})}
2018-09-25 18:43:12,406 - pwatcher.blocking:367 - DEBUG - Basic submitter: StringJobSubmitter('qsub -S /bin/bash -
sync y -V  \\\n-q ${JOB_QUEUE}     \\\n-N ${JOB_NAME}      \\\n-o "${JOB_STDOUT}"  \\\n-e "${JOB_STDERR}"  \\\n-pe
 smp ${NPROC}    \\\n-l h_vmem=${MB}M    \\\n"${JOB_SCRIPT}"')
2018-09-25 18:43:12,406 - pwatcher.blocking:370 - DEBUG -  starting job Job(jobid='Pa93d51772c1627', cmd='/bin/bas
h run.sh', rundir='/lustre1/gd98309/funzip/ecoli3/0-rawreads/build', options={'JOB_QUEUE': 'default', 'pwatcher_ty
pe': 'blocking', 'use_tmpdir': False, 'MB': 4000, 'job_type': 'local', 'submit': 'qsub -S /bin/bash -sync y -V  \\
\n-q ${JOB_QUEUE}     \\\n-N ${JOB_NAME}      \\\n-o "${JOB_STDOUT}"  \\\n-e "${JOB_STDERR}"  \\\n-pe smp ${NPROC}
    \\\n-l h_vmem=${MB}M    \\\n"${JOB_SCRIPT}"', 'NPROC': 1, 'njobs': '32', 'local': 0})
2018-09-25 18:43:12,406 - pwatcher.blocking:212 - DEBUG - Writing wrapper "/lustre1/gd98309/funzip/ecoli3/0-rawrea
ds/build/run-Pa93d51772c1627.bash"
2018-09-25 18:43:16,368 - pwatcher.blocking:322 - DEBUG - env_extra={'PYPEFLOW_JOB_START_SCRIPT': '/lustre1/gd9830
9/funzip/ecoli3/0-rawreads/build/run-Pa93d51772c1627.bash',
 'PYPEFLOW_JOB_START_TIMEOUT': '60'}
2018-09-25 18:43:16,369 - pwatcher.blocking:82 - DEBUG - Thread notify_started(Pa93d51772c1627).
2018-09-25 18:43:16,369 - pypeflow.simple_pwatcher_bridge:153 - DEBUG - Result of watcher.run()={'submitted': ['Pa
93d51772c1627']}
2018-09-25 18:43:16,369 - pwatcher.blocking:223 - DEBUG - hello! started Thread <JobThread(Thread-1, started 47580
893337344)>
2018-09-25 18:43:16,369 - pypeflow.simple_pwatcher_bridge:324 - DEBUG - N in queue: 1 (max_jobs=32)
2018-09-25 18:43:16,370 - pwatcher.blocking:463 - DEBUG - query(which='list', jobids=<1>)
2018-09-25 18:43:16,370 - pwatcher.blocking:227 - INFO - Popen: 'qsub -S /bin/bash -sync y -V  \
-q default     \
-N Pa93d51772c1627      \
-o "/lustre1/gd98309/funzip/ecoli3/0-rawreads/build/run-Pa93d51772c1627.bash.stdout"  \
-e "/lustre1/gd98309/funzip/ecoli3/0-rawreads/build/run-Pa93d51772c1627.bash.stderr"  \
-pe smp 1    \
-l h_vmem=4000M    \
"/home/gd98309/.conda/envs/falconda/lib/python2.7/site-packages/pwatcher/mains/job_start.sh"'
2018-09-25 18:43:16,370 - pypeflow.simple_pwatcher_bridge:334 - INFO - (slept for another 0.0s -- another 1 loop i
terations)
2018-09-25 18:43:16,375 - pwatcher.blocking:232 - DEBUG - pid: 54501
2018-09-25 18:43:16,389 - pwatcher.blocking:235 - DEBUG - rc: 1
2018-09-25 18:43:16,389 - pwatcher.blocking:87 - DEBUG - Thread notify_exited(Pa93d51772c1627->1).
2018-09-25 18:43:16,473 - pypeflow.simple_pwatcher_bridge:324 - DEBUG - N in queue: 1 (max_jobs=32)
2018-09-25 18:43:16,473 - pwatcher.blocking:463 - DEBUG - query(which='list', jobids=<1>)
2018-09-25 18:43:16,473 - pypeflow.simple_pwatcher_bridge:94 - ERROR - Task Node(0-rawreads/build) failed with exi
t-code=1
2018-09-25 18:43:16,473 - pypeflow.simple_pwatcher_bridge:342 - DEBUG - recently_done: [(Node(0-rawreads/build), F
alse)]
2018-09-25 18:43:16,474 - pypeflow.simple_pwatcher_bridge:343 - DEBUG - Num done in this iteration: 1
2018-09-25 18:43:16,474 - pypeflow.simple_pwatcher_bridge:357 - ERROR - Some tasks are recently_done but not satis
fied: set([Node(0-rawreads/build)])
2018-09-25 18:43:16,474 - pypeflow.simple_pwatcher_bridge:358 - ERROR - ready: set([])
	submitted: set([])
2018-09-25 18:43:16,474 - pwatcher.blocking:467 - DEBUG - delete(which='known', jobids=<0>)
2018-09-25 18:43:16,474 - pwatcher.blocking:431 - ERROR - Noop. We cannot kill blocked threads. Hopefully, everyth
ing will die on SIGTERM.
2018-09-25 18:43:16,474 - pypeflow.simple_pwatcher_bridge:189 - DEBUG - In notifyTerminate(), result of delete:None

[armintoepfer]

Thank you, we'll look into it.

[gconcepcion]

Hi @gbdias

Your run is failing in the initial step during the building of the read database.

The actual error would be found here:
0-rawreads/build/run-*.bash.stderr

Were you successful running the test example here?
https://github.com/PacificBiosciences/pb-assembly#example-data-set

It's generally better to ensure the test dataset works prior to running your own data to rule out installation problems.

[gbdias]

Hi guys,
Thanks a lot for the quick reply.

• I suspect this has to do with resource allocation.

• I just ran the test dataset with the same script but with the test cfg file from the FALCON-examples repo and it completed both Falcon and Unzip.

• Test cfg file from the greg run:

[General]
#sge_option_da =
#sge_option_la =
#sge_option_pda =
#sge_option_pla =
#sge_option_cns =
#sge_option_fc =
#pwatcher_type = blocking
#job_type = string
#job_queue = bash -C ${CMD} >| ${STDOUT_FILE} 2>| ${STDERR_FILE}

# list of files of the initial bas.h5 files
input_fofn = input.fofn
#input_fofn = preads.fofn

input_type = raw
#input_type = preads

#openending = True

# The length cutoff used for seed reads used for initial mapping
length_cutoff = 1000
genome_size = 200000
#seed_coverage = 60

# The length cutoff used for seed reads usef for pre-assembly
length_cutoff_pr = 1

#pa_REPmask_code = 1,300;0,300;0,300
pa_daligner_option   = -e.8 -l3200 -k18 -h480  -w8 -s100
ovlp_daligner_option = -e.9 -l2500 -k24 -h1024 -w6 -s100
pa_HPCdaligner_option   =  -v -B128 -M24
ovlp_HPCdaligner_option =  -v -B128 -M24
#pa_HPCTANmask_option = -k18 -h480 -w8 -e.8 -s100
#pa_HPCREPmask_option = -k18 -h480 -w8 -e.8 -s100

pa_DBsplit_option = -a -x500 -s4
ovlp_DBsplit_option = -s100

falcon_sense_option = --output-multi --min-cov-aln 4 --min-idt 0.70 --min-cov 4 --max-n-read 200 --n-core 8
falcon_sense_skip_contained = False

overlap_filtering_setting = --max-diff 120 --max-cov 120 --min-cov 2 --n-core 8
#dazcon = 1

[job.defaults]
job_type = local

#use_tmpdir = /scratch
pwatcher_type = blocking
job_type = string
submit = bash -C ${CMD} >| ${STDOUT_FILE} 2>| ${STDERR_FILE}
#job_queue = bash -C ${CMD}
# By dropping STD*_FILE, we see all output on the console.
# That helps debugging in TravisCI/Bamboo.

NPROC=4
#njobs=1
[job.step.da]
[job.step.pda]
[job.step.la]
#NPROC=1
[job.step.pla]
#NPROC=1
[job.step.cns]
#njobs=1
NPROC=8
[job.step.asm]
NPROC=32

• My cfg file for E. coli

#### Input
[General]
input_fofn=input.fofn
input_type=raw
pa_DBdust_option=True
pa_fasta_filter_option=pass
target=assembly
skip_checks=True
LA4Falcon_preload=false

#### Data Partitioning
pa_DBsplit_option=-x500 -s200
ovlp_DBsplit_option=-x500 -s200

#### Repeat Masking
pa_HPCTANmask_option=
pa_REPmask_code=0,300;0,300;0,300

####Pre-assembly
genome_size=4600000
seed_coverage=20
length_cutoff=-1 
pa_HPCdaligner_option=-v -B128 -M24
pa_daligner_option=-e.8 -l2000 -k18 -h480  -w8 -s100
falcon_sense_option=--output-multi --min-idt 0.70 --min-cov 2 --max-n-read 400
falcon_sense_greedy=False

####Pread overlapping
ovlp_daligner_option=-e.93 -l2500 -k24 -h1024 -w6 -s100
ovlp_HPCdaligner_option=-v -B128 -M24

####Final Assembly
overlap_filtering_setting=--max-diff 100 --max-cov 100 --min-cov 2
fc_ovlp_to_graph_option=
length_cutoff_pr=1000

[job.defaults]
job_type=local
pwatcher_type=blocking
JOB_QUEUE=default
MB=32768
NPROC=8
njobs=32
submit = qsub -S /bin/bash -sync y -V  \
  -q ${JOB_QUEUE}     \
  -N ${JOB_NAME}      \
  -o "${JOB_STDOUT}"  \
  -e "${JOB_STDERR}"  \
  -pe smp ${NPROC}    \
  -l h_vmem=${MB}M    \
  "${JOB_SCRIPT}"

[job.step.da]
# NPROC=4
# MB=32768
# njobs=240
[job.step.la]
# NPROC=4
# MB=32768
# njobs=240
[job.step.cns]
NPROC=4
MB=49152
njobs=2
[job.step.pda]
# NPROC=4
# MB=32768
# njobs=240
[job.step.pla]
# NPROC=4
# MB=32768
# njobs=240
[job.step.asm]
NPROC=4
MB=32768
njobs=3

• I am having a hard time figuring out how should I scale it up on a bigger machine, but in local mode (not sge). Do you guys have any recommendation on how to do this?

Thanks,
Guilherme

[gbdias]

Hi Greg,

My fc_run.cfg is the last one I just posted. I though it was in local mode cause I used the job_type = local flag.
Do I also need the job_type = string and submit = bash -C ${CMD} >| ${STDOUT_FILE} 2>| ${STDERR_FILE} for local mode?

[gconcepcion]

@gbdias Ahh.

The greg200k-sv2 example was run in local mode w/ a local mode submit string.
You ran the ecoli job in local mode w/ an sge submit string.

For local mode, you need to run w/ the local mode submit line:
bash -C ${CMD} >| ${STDOUT_FILE} 2>| ${STDERR_FILE}
See also the Job Distribution section here:
https://github.com/PacificBiosciences/pb-assembly#configuration

[gbdias]

Okie dokie.
I will fix this and try again!
Thanks

[wuxiaopei0509]

I have the similar problem at 0-rawreads/cns-runs stage and I check the run-Pec6a2970c36c74.bash.stderr .I found the job was killed nonarbitrary;the error info as follows:

2019-09-27 14:12:07,314 - root - CRITICAL - Error in /data/wuxiaopei/conda/envs/dujuan_denovo_pb-assembly0.0.4/lib/python2.7/site-packages/
pypeflow/do_task.py
with args="{'json_fn': '/1data/wuxiaopei/zhejiangdaxue_dujuan/Falcon_assembly/0-rawreads/cns-runs/cns_00043/task.json',\n 'timeout': 30,\n 'tmpdir': None}"

Could y help me with it? Thanks a lot.

[pb-cdunn]

Hmmm. If there is nothing in stderr, my best guess is that your job-distributor killed it for exceeding some resource limits.