epi2me-labs / wf-metagenomics Goto Github PK
View Code? Open in Web Editor NEWMetagenomic classification of long-read sequencing data
License: Other
wf-metagenomics's People
Contributors
Stargazers
Watchers
Forkers
bnwaweru flavianhm shelbyreimer yashasdevasurmutt hindlem gagayuan aamcgenomics robsyme eparisis cnyjwang xprize-msu sarah-buddle xavier-j mariofajardo thanhleviet samanthaeva conanyangqun pythseq odielwoolmore salome-brunonwf-metagenomics's Issues
ERROR: Validation of pipeline parameters failed!
Hi,
when I wanted to test the metagenomics workflow, I received following error message:
ERROR: Validation of pipeline parameters failed!
- --sources: expected type: JSONObject, found: String ([TARGLOCI:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-ribosomal-survey/targeted_loci/ncbi_targeted_loci_16s_18s.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-ribosomal-survey/targeted_loci/ncbi_targeted_loci_16s_18s.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-ribosomal-survey/targeted_loci/ncbi_targeted_loci_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-ribosomal-survey/targeted_loci/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-ribosomal-survey/targeted_loci/ref2taxid.targloci.tsv, taxonomy:https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz]])
One of the commands I used was the following:
nextflow run epi2me-labs/wf-metagenomics --fastq ./basecalling/pass/ --out_dir nextflow-output/minimap/ --minimap2 --report_name minimap --threads 20
I used some other variations of the command too, but all of them returned this error message
Kind regards,
Hans-Peter
Unable to run pipeline on test or any other data
I cannot successfully run the pipeline on my own or the test data. Both kraken2 and minimap2 fail to classify any sequences.
I downloaded the test data and confirmed that it contains 1000 reads. Then run the pipeline as follows:
#!/usr/bin/env bash
set -euo pipefail
export NXF_VER='21.10.6'
pipeline='https://github.com/epi2me-labs/wf-metagenomics'
nextflow pull $pipeline
nextflow -c 'local.config' run \
$pipeline \
-r 'master' \
-params-file 'params.yml' \
-resumeThe local config just contains tower.enabled = false.
params.yml
fastq: "/.../data/reads.fastq.gz"
minimap2: true
kraken2: true
threads: 30When I run like this, kraken2 is the first to fail.
Error executing process > 'pipeline:kraken2 (1)'
Caused by:
Missing output file(s) `*.classified.fastq` expected by process `pipeline:kraken2 (1)`
Command executed:
kraken2 --db database_dir --threads 30 --report reads.kraken2_report.txt --classified-out reads.kraken2.classified.fastq --unclassified-out reads.kr
aken2.unclassified.fastq reads.fastq > reads.kraken2.assignments.tsv
awk -F '\t' '{print $3}' reads.kraken2.assignments.tsv > taxids.tmp
taxonkit --data-dir taxonomy_dir lineage -R taxids.tmp | aggregate_lineages.py -p reads.kraken2
Command exit status:
0
Command output:
(empty)
Command error:
Loading database information... done.
0 sequences (0.00 Mbp) processed in 0.131s (0.0 Kseq/m, 0.00 Mbp/m).
0 sequences classified (-nan%)
0 sequences unclassified (-nan%)
When I set kraken2 to false, the minimap2 step completes but the report fails.
Error executing process > 'pipeline:makeReport'
Caused by:
Process `pipeline:makeReport` terminated with an error exit status (1)
Command executed:
report.py wf-metagenomics-report.html --versions versions --params params.json --summaries reads.stats --lineages reads.minimap2.lineages.json --vistempl report-visualisation.html
Command exit status:
1
Command output:
(empty)
Command error:
Traceback (most recent call last):
File "/home/moritz/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/report.py", line 116, in <module>
main()
File "/home/moritz/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/report.py", line 99, in main
section=fastcat.full_report(
File "/home/epi2melabs/conda/lib/python3.8/site-packages/aplanat/components/fastcat.py", line 140, in full_report
read_length = read_length_plot(stats, min_len=min_len, max_len=max_len)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/aplanat/components/fastcat.py", line 29, in read_length_plot
mean_length = total_bases / len(seq_summary)
ZeroDivisionError: division by zero
This last error is a bit suspicious since the pipeline runs with the standard profile and thus Docker...
I also tried to run it with revision v1.1.4 and nextflow 22.04.3 same errors.
[Bug]: How to specify Bracken length
What happened?
Please can I ask how you suggest choosing the read length option for Bracken?
Bracken itself wants a set read length - and obviously with nanopore the read lengths vary considerably.
It seems that this is a live issue with the Bracken development team: jenniferlu717/Bracken#60
It seems that a minimum read length might be the safest option?
Many thanks,
Jack
Operating System
ubuntu 18.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
Singularity
Workflow Version
v2.0.8-g3e8fdcd
Relevant log output
na[Bug]: Cannot run epi2me-labs/wf-metagenomics offline
What happened?
We need to be able to run epi2me workflows offline on the command line
Config works if I set http_proxy and https_proxy env variables. If I execute a new iteration of the workflow of a previous run (with a new output folder), the workflow still tries to download database and tax files. I've already tried to set --database and --taxonomy or --store_dir but it still fails.
Syntax:
nextflow run epi2me-labs/wf-metagenomics --fastq reads_filtered.fastq.gz --batch_size 1000 --out_dir output2 --min_len 200 --min_read_qual 8 --threads 2 --database store_dir/ncbi_targeted_loci_kraken2/database_dir/ --taxonomy store_dir/taxdmp_2023-01-01/taxonomy_dir/ --store_dir store_dir/
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
Docker
Workflow Version
This is epi2me-labs/wf-metagenomics v2.2.1-g5344ddc.
Relevant log output
N E X T F L O W ~ version 23.04.1
Launching `https://github.com/epi2me-labs/wf-metagenomics` [shrivelled_aryabhata] DSL2 - revision: 5344ddcd2a [master]
WARN: NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
|||||||||| _____ ____ ___ ____ __ __ _____ _ _
|||||||||| | ____| _ \_ _|___ \| \/ | ____| | | __ _| |__ ___
||||| | _| | |_) | | __) | |\/| | _| _____| |/ _` | '_ \/ __|
||||| | |___| __/| | / __/| | | | |__|_____| | (_| | |_) \__ \
|||||||||| |_____|_| |___|_____|_| |_|_____| |_|\__,_|_.__/|___/
|||||||||| wf-metagenomics v2.2.1-g5344ddc
--------------------------------------------------------------------------------
Core Nextflow options
revision : master
runName : shrivelled_aryabhata
containerEngine: docker
launchDir : /home/fmg/Documents/analysis/bats
workDir : /home/fmg/Documents/analysis/bats/work
projectDir : /home/fmg/.nextflow/assets/epi2me-labs/wf-metagenomics
userName : fmg
profile : standard
configFiles : /home/fmg/.nextflow/assets/epi2me-labs/wf-metagenomics/nextflow.config
Input Options
fastq : reads_filtered.fastq.gz
batch_size : 1000
Reference Options
database : store_dir/ncbi_targeted_loci_kraken2/database_dir/
taxonomy : store_dir/taxdmp_2023-01-01/taxonomy_dir/
store_dir : store_dir/
database_sets : [ncbi_16s_18s:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ref2taxid.targloci.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip], ncbi_16s_18s_28s_ITS:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ref2taxid.ncbi_16s_18s_28s_ITS.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip], PlusPF-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspf_08gb_20230314.tar.gz, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/new_taxdump_2023-03-01.zip], PlusPFP-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspfp_08gb_20230314.tar.gz, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/new_taxdump_2023-03-01.zip]]
Output Options
out_dir : output2
Advanced Options
min_len : 200
min_read_qual : 8
Other parameters
process_label : wfmetagenomics
!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-metagenomics for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
--------------------------------------------------------------------------------
This is epi2me-labs/wf-metagenomics v2.2.1-g5344ddc.
--------------------------------------------------------------------------------
Checking inputs.
Note: Memory available to the workflow must be slightly higher than size of the database index
Note: Memory available to the workflow must be slightly higher than size of the database index
Checking custom taxonomy mapping exists
Checking custom kraken2 database exists
Checking fastq input.
executor > local (4)
executor > local (4)
[- ] process > fastcat (1) -
[- ] process > kraken_pipeline:unpackTaxonomy -
[- ] process > kraken_pipeline:unpackDatabase -
[- ] process > kraken_pipeline:determine_bracken_length -
[- ] process > kraken_pipeline:kraken_server -
[- ] process > kraken_pipeline:rebatchFastq -
[- ] process > kraken_pipeline:kraken2_client -
[- ] process > kraken_pipeline:progressive_stats -
[- ] process > kraken_pipeline:progressive_kraken_reports -
[- ] process > kraken_pipeline:progressive_bracken -
[- ] process > kraken_pipeline:getVersions -
[- ] process > kraken_pipeline:getParams -
[- ] process > kraken_pipeline:makeReport -
[- ] process > kraken_pipeline:output -
[- ] process > kraken_pipeline:stop_kraken_server -
WARN: Unable to stage foreign file: https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib (try 1) -- Cause: Connection refused
WARN: Unable to stage foreign file: https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib (try 2) -- Cause: Connection refused
WARN: Unable to stage foreign file: https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib (try 3) -- Cause: Connection refused
ERROR ~ Error executing process > 'kraken_pipeline:unpackDatabase'
Caused by:
Can't stage file https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib -- reason: Connection refused
Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
-- Check '.nextflow.log' file for details
WARN: Graphviz is required to render the execution DAG in the given format -- See http://www.graphviz.org for more info.[Bug]: workflow fails when more 28 fastq files are being analyzed
What happened?
The workflow aborts due to unexpected error when more than 28 fastq files are being analyzed. I have avoided this by concatenating the files (>400) first and then running the workflow. This workaround only works on post run analysis. Real time analysis, with the --watch_path flag, fails on the 29th fastq file that is placed in the watch path.
command:
nextflow run epi2me-labs/wf-metagenomics --fastq /home/Desktop/Test_data_cat --out_dir /home/Data_Analysis/a_b_s/gsup_5.0.16/Analysis/BC_both_ends/wf-metagenomics/k2_pfp_16_BC18.29_v2.0.6_1 --database /home/Data_DB/Kraken2_db/k2_pluspfp_16gb_20220908
Error:
This is epi2me-labs/wf-metagenomics v2.0.6-gf5d1e16.
Checking inputs.
Checking custom kraken2 database exists
[- ] process > kraken_pipeline:unpackTaxonomy -
executor > local (3)
executor > local (3)
executor > local (3)
executor > local (4)
executor > local (4)
executor > local (4)
executor > local (4)
executor > local (4)
executor > local (4)
executor > local (7)
executor > local (7)
executor > local (7)
executor > local (8)
executor > local (10)
executor > local (10)
executor > local (10)
executor > local (13)
executor > local (14)
executor > local (14)
executor > local (17)
executor > local (17)
executor > local (17)
executor > local (17)
executor > local (22)
executor > local (22)
executor > local (25)
executor > local (25)
executor > local (181)
[skipped ] process > kraken_pipeline:unpackTaxonomy [100%] 1 of 1, stored: 1 ✔
[skipped ] process > kraken_pipeline:unpackDatabase [100%] 1 of 1, stored: 1 ✔
[7b/f79621] process > kraken_pipeline:determine_b... [100%] 1 of 1 ✔
[8b/3864e5] process > kraken_pipeline:kraken_server [100%] 1 of 1 ✔
[68/7222bf] process > kraken_pipeline:kraken2_cli... [100%] 30 of 30 ✔
[a8/db9b9b] process > kraken_pipeline:progressive... [100%] 30 of 30 ✔
[f0/0264b1] process > kraken_pipeline:progressive... [100%] 30 of 30 ✔
[d0/8682e0] process > kraken_pipeline:bracken (28) [100%] 28 of 28
[52/116b2b] process > kraken_pipeline:getVersions [100%] 1 of 1 ✔
[81/a91a62] process > kraken_pipeline:getParams [100%] 1 of 1 ✔
[ee/a0a1b6] process > kraken_pipeline:makeReport ... [100%] 28 of 28
[2b/da7d81] process > kraken_pipeline:output (30) [100%] 30 of 30
[a6/e59e7a] process > kraken_pipeline:stop_kraken... [100%] 1 of 1 ✔
Execution aborted due to an unexpected error
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
Docker
Workflow Version
v2.0.6-gf5d1e16
Relevant log output
none[Bug]: kraken2_client fails with the test data. Server is error state. Return code 12
What happened?
OS: Ubuntu 22
The problem also happened when trying to run the workflow from the EPI2ME Labs desktop application, and also using my own datasets.
Thanks in advance!
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - Execution Profile
Conda
Workflow Version
v2.0.3
Relevant log output
/home/fume/opt/miniconda3/envs/epi2melabs-wf-metagenomics/bin/nextflow run main.nf --fastq test_data/ PlusPF-8 --out_dir /home/fume/test --kraken2 --bracken_level G
N E X T F L O W ~ version 22.10.1
Launching `main.nf` [reverent_wing] DSL2 - revision: c82209066b
WARN: NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASE
Core Nextflow options
runName : reverent_wing
containerEngine: docker
launchDir : /home/fume/opt/wf-metagenomics
workDir : /home/fume/opt/wf-metagenomics/work
projectDir : /home/fume/opt/wf-metagenomics
userName : fume
profile : standard
configFiles : /home/fume/opt/wf-metagenomics/nextflow.config
Core options
fastq : test_data/
out_dir : /home/fume/test
store_dir : store_dir
sources : [ncbi_16s_18s:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ref2taxid.targloci.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz], ncbi_16s_18s_28s_ITS:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ref2taxid.ncbi_16s_18s_28s_ITS.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz], PlusPF-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspf_8gb_20210517.tar.gz, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz]]
Kraken2 options
bracken_level : G
Generic options
threads : 4
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use epi2me-labs/wf-metagenomics for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
Checking inputs.
executor > local (7)
[- ] process > kraken_pipeline:unpackTaxonomy -
[skipped ] process > kraken_pipeline:unpackDatabase [100%] 1 of 1, stored: 1 ✔
[b8/b5edd2] process > kraken_pipeline:kraken_server [100%] 1 of 1 ✔
[39/7e1e31] process > kraken_pipeline:combineFilterFastq (6) [ 11%] 1 of 9
[bf/0fcfe1] process > kraken_pipeline:progressiveStats (1) [ 0%] 0 of 1
[8d/f524e9] process > kraken_pipeline:kraken2_client (1) [100%] 1 of 1, failed: 1
[- ] process > kraken_pipeline:progressive_kreports -
[- ] process > kraken_pipeline:taxon_kit -
[- ] process > kraken_pipeline:bracken -
[e9/81c44f] process > kraken_pipeline:getVersions [100%] 1 of 1 ✔
[98/e5557b] process > kraken_pipeline:getParams [100%] 1 of 1 ✔
[- ] process > kraken_pipeline:makeReport -
[- ] process > kraken_pipeline:catAssignmentsprogressive -
[- ] process > kraken_pipeline:stop_kraken_server -
[- ] process > kraken_pipeline:output -
[- ] process > kraken_pipeline:output_dir -
[- ] process > output -
Input directory assumed to be containing one or more directories containing fastq files.
Staging foreign file: https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz
[skipping] Stored process > kraken_pipeline:unpackDatabase
Staging foreign file: https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz
Error executing process > 'kraken_pipeline:kraken2_client (1)'
Caused by:
Process `kraken_pipeline:kraken2_client (1)` terminated with an error exit status (12)
Command executed:
kraken2_client --port 8080 --report report.txt --sequence "barcode01.6.fastq.gz" > "barcode01.kraken2.assignments.tsv"
tail -n +1 report.txt > "barcode01.kraken2.report.txt"
Command exit status:
12
Command output:
(empty)
Command error:
Connecting to server: localhost:8080.
Classifying sequence stream.
Server is in error state:
Return code: 12
Work dir:
/home/fume/opt/wf-metagenomics/work/8d/f524e93851264c1befe6fe5fb7b01a
Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
WARN: Killing running tasks (2)
WARN: Failed to render execution report -- see the log file for details
WARN: Failed to render execution timeline -- see the log file for details[Bug]: Cannot get workflow to run
What happened?
System - Ubuntu 18.04 with Nextflow 22.10.0 and wf-metagenomics v2.0.0 (b2bd2b8)
Command - nextflow run epi2me-labs/wf-metagenomics --fastq /home/data1/Analyzed_data/metactrrun2/guppy_6.1.7/demux_trim/BC96_ZymoStd.fastq.gz --kraken2 --threads 20
Output - hangs on process > kraken_pipeline:kraken_server (left running for 24hrs) and kraken2_client never starts.
Checking inputs.
executor > local (7)
[d8/3d42fa] process > kraken_pipeline:unpackTaxonomy [100%] 1 of 1 ✔
[2f/ca0892] process > kraken_pipeline:unpackDatabase [100%] 1 of 1 ✔
[b8/759512] process > kraken_pipeline:kraken_server [ 0%] 0 of 1
[- ] process > kraken_pipeline:combineFilterFastq -
[- ] process > kraken_pipeline:progressiveStats -
[- ] process > kraken_pipeline:kraken2_client -
[- ] process > kraken_pipeline:progressive_kreports -
[- ] process > kraken_pipeline:taxon_kit -
[- ] process > kraken_pipeline:bracken -
[fd/a60b1a] process > kraken_pipeline:getVersions [100%] 1 of 1 ✔
[38/ed71ac] process > kraken_pipeline:getParams [100%] 1 of 1 ✔
[- ] process > kraken_pipeline:makeReport -
[- ] process > kraken_pipeline:mergeclassifiedProgressive -
[- ] process > kraken_pipeline:mergeunclassifiedProgressive -
[- ] process > kraken_pipeline:catAssignmentsprogressive -
[- ] process > kraken_pipeline:stop_kraken_server -
[- ] process > kraken_pipeline:output -
[- ] process > kraken_pipeline:output_dir -
[2f/14c7e9] process > output (1) [100%] 2 of 2 ✔
I have also tried this workflow on a different system running Ubuntu 20.04. The workflow failed regardless if it was executed with Docker or Conda.
Operating System
ubuntu 18.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - Execution Profile
Docker
Workflow Version
Relevant log output
.command.log from b8/759512
Loading database information... done.
Server listening on localhost:8080. Press Ctrl-C to end.
Not sure what other logs would be helpful as other log files are empty.I have also tried EPI2ME Labs (v3.15) with Labs environment v1.2.5. The workflow was terminated due to the following error:
Checking epi2me-labs/wf-metagenomics ...
epi2me-labs/wf-metagenomics contains uncommitted changes -- cannot pull from repository
N E X T F L O W ~ version 22.04.0
Project epi2me-labs/wf-metagenomics contains uncommitted changes -- Cannot switch to revision: v1.1.4
[Bug]: kraken_pipeline.nf - kraken2_client could not get any data from kraken2_server when using a big kraken2 index
What happened?
On kraken_pipeline.nf, if the kraken index is big (example: nt), the kraken_server process do not have enough time to load de index and kraken_client process fails.
Solution:
Change ** kraken2_client process** like this:
process kraken2_client {
errorStrategy { sleep(30000); return 'retry' }
maxRetries 999
label "wfmetagenomics"
containerOptions {workflow.profile != "singularity" ? "--network host" : ""}
maxForks 1
input:
tuple val(sample_id), path(reads)
output:
tuple val(sample_id), path("*kraken2_report.txt"), path(reads), path("*.tsv")
script:
"""
kraken2_client --port $params.port --sequence "${reads}" > "${sample_id}.kraken2.assignments.tsv"
kraken2_client --port $params.port --report --sequence "${reads}" > "out.txt"
tail -n +2 "out.txt" > "tmp.txt"
head -n -6 "tmp.txt" > "${sample_id}.kraken2_report.txt"
"""
}
errorStrategy was set to retry every 30 sec. This avoid a lot of processes being started and failing during the kraken2 database loading, preventing the system to overload.
maxRetries was set to 999 to preven
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
ontresearch/wf-metagenomics
Workflow Execution - Execution Profile
Docker
Workflow Version
v2.0.1
Relevant log output
lost the logs because already solved the problem in my local repo. The pipeline failed in kraken2_client nextflow process telling that could not get any data from kraken2_server.[Bug]: Permission denied, I have tried almost everything from another similar issue, also tried to give permissions using chmod, didn't work.
What happened?
A bug happened!
Operating System
ubuntu 20.04
Workflow Execution
EPI2ME Labs desktop application
Workflow Execution - EPI2ME Labs Versions
EPI2ME Labs V4.1.3
Workflow Execution - CLI Execution Profile
Docker
Workflow Version
v2.0.10
Relevant log output
Mar-17 10:36:17.078 [main] DEBUG nextflow.cli.Launcher - $> /usr/lib/epi2melabs/resources/nextflow-all -log /home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/nextflow.log run /home/abiplantlab/epi2melabs/workflows/epi2me-labs/wf-metagenomics/main.nf -params-file /home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/params.json -w /home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/work -ansi-log false -offline -profile standard -resume -name reverent_joliot -c /home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/demo.config
Mar-17 10:36:17.143 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 22.04.5
Mar-17 10:36:17.161 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /home/abiplantlab/epi2melabs/workflows/epi2me-labs/wf-metagenomics/nextflow.config
Mar-17 10:36:17.164 [main] DEBUG nextflow.config.ConfigBuilder - User config file: /home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/demo.config
Mar-17 10:36:17.164 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/abiplantlab/epi2melabs/workflows/epi2me-labs/wf-metagenomics/nextflow.config
Mar-17 10:36:17.164 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/demo.config
Mar-17 10:36:17.197 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Mar-17 10:36:17.942 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Mar-17 10:36:17.995 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion
Mar-17 10:36:18.012 [main] INFO nextflow.cli.CmdRun - Launching `/home/abiplantlab/epi2melabs/workflows/epi2me-labs/wf-metagenomics/main.nf` [reverent_joliot] DSL2 - revision: db65d066e8
Mar-17 10:36:18.025 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; plugins-dir=/home/abiplantlab/epi2melabs/workflows/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Mar-17 10:36:18.026 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Mar-17 10:36:18.038 [main] INFO org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Mar-17 10:36:18.039 [main] INFO org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Mar-17 10:36:18.043 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Mar-17 10:36:18.055 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Mar-17 10:36:18.111 [main] DEBUG nextflow.Session - Session uuid: 6213c586-62d6-419f-a4d4-58225e8b7a9f
Mar-17 10:36:18.111 [main] DEBUG nextflow.Session - Run name: reverent_joliot
Mar-17 10:36:18.112 [main] DEBUG nextflow.Session - Executor pool size: 12
Mar-17 10:36:18.145 [main] DEBUG nextflow.cli.CmdRun -
Version: 22.04.5 build 5709
Created: 15-07-2022 16:22 UTC (11:22 CDT)
System: Linux 5.15.0-67-generic
Runtime: Groovy 3.0.10 on OpenJDK 64-Bit Server VM 11.0.18+10-post-Ubuntu-0ubuntu120.04.1
Encoding: UTF-8 (UTF-8)
Process: 148054@abiplantlab-OptiPlex-3070 [127.0.1.1]
CPUs: 12 - Mem: 15.4 GB (470.8 MB) - Swap: 2 GB (973.2 MB)
Mar-17 10:36:18.175 [main] DEBUG nextflow.Session - Work-dir: /home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/work [ext2/ext3]
Mar-17 10:36:18.221 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[GoogleLifeSciencesExecutor, AwsBatchExecutor]
Mar-17 10:36:18.234 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Mar-17 10:36:18.254 [main] DEBUG nextflow.Session - Observer factory: TowerFactory
Mar-17 10:36:18.284 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Mar-17 10:36:18.293 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 13; maxThreads: 1000
Mar-17 10:36:18.452 [main] DEBUG nextflow.Session - Session start invoked
Mar-17 10:36:18.457 [main] DEBUG nextflow.trace.TraceFileObserver - Flow starting -- trace file: /home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/output/execution/trace.txt
Mar-17 10:36:18.461 [main] DEBUG nextflow.Session - Using default localLib path: /home/abiplantlab/epi2melabs/workflows/epi2me-labs/wf-metagenomics/lib
Mar-17 10:36:18.465 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/abiplantlab/epi2melabs/workflows/epi2me-labs/wf-metagenomics/lib
Mar-17 10:36:18.466 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/abiplantlab/epi2melabs/workflows/epi2me-labs/wf-metagenomics/lib/nfcore_external_java_deps.jar
Mar-17 10:36:19.579 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Mar-17 10:36:20.886 [main] WARN nextflow.NextflowMeta$Preview - NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASE
Mar-17 10:36:20.993 [main] INFO nextflow.Nextflow -
�[0;92m|||||||||| �[0m�[2m_____ ____ ___ ____ __ __ _____ _ _
�[0;92m|||||||||| �[0m�[2m| ____| _ \_ _|___ \| \/ | ____| | | __ _| |__ ___
�[0;33m||||| �[0m�[2m| _| | |_) | | __) | |\/| | _| _____| |/ _` | '_ \/ __|
�[0;33m||||| �[0m�[2m| |___| __/| | / __/| | | | |__|_____| | (_| | |_) \__ \
�[0;94m|||||||||| �[0m�[2m|_____|_| |___|_____|_| |_|_____| |_|\__,_|_.__/|___/
�[0;94m|||||||||| �[0m�[1mwf-metagenomics v2.0.10�[0m
�[2m--------------------------------------------------------------------------------�[0m
�[1mCore Nextflow options�[0m
�[0;34mrunName : �[0;32mreverent_joliot�[0m
�[0;34mcontainerEngine: �[0;32mdocker�[0m
�[0;34mlaunchDir : �[0;32m/home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8�[0m
�[0;34mworkDir : �[0;32m/home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/work�[0m
�[0;34mprojectDir : �[0;32m/home/abiplantlab/epi2melabs/workflows/epi2me-labs/wf-metagenomics�[0m
�[0;34muserName : �[0;32mabiplantlab�[0m
�[0;34mprofile : �[0;32mstandard�[0m
�[0;34mconfigFiles : �[0;32m/home/abiplantlab/epi2melabs/workflows/epi2me-labs/wf-metagenomics/nextflow.config, /home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/demo.config�[0m
�[1mInput Options�[0m
�[0;34mfastq : �[0;32m/home/abiplantlab/epi2melabs/demo/epi2me-labs/wf-metagenomics/wf-metagenomics-demo/test_data�[0m
�[1mReference Options�[0m
�[0;34mdatabase_sets : �[0;32m[ncbi_16s_18s:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ref2taxid.targloci.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip], ncbi_16s_18s_28s_ITS:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ref2taxid.ncbi_16s_18s_28s_ITS.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip], PlusPF-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspf_08gb_20221209.tar.gz, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2022-09-01.zip]]�[0m
�[1mOutput Options�[0m
�[0;34mout_dir : �[0;32m/home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/output�[0m
�[1mOther parameters�[0m
�[0;34mprocess_label : �[0;32mwfmetagenomics�[0m
!! Only displaying parameters that differ from the pipeline defaults !!
�[2m--------------------------------------------------------------------------------�[0m
If you use epi2me-labs/wf-metagenomics for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
�[2m--------------------------------------------------------------------------------�[0m
This is epi2me-labs/wf-metagenomics v2.0.10.
�[2m--------------------------------------------------------------------------------�[0m
Mar-17 10:36:21.842 [main] INFO nextflow.Nextflow - Checking inputs.
Mar-17 10:36:21.893 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name kraken_pipeline:unpackTaxonomy
Mar-17 10:36:21.909 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-17 10:36:21.909 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-17 10:36:21.912 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Mar-17 10:36:21.916 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=8; memory=8 GB; capacity=12; pollInterval=100ms; dumpInterval=5m
Mar-17 10:36:21.994 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name kraken_pipeline:unpackDatabase
Mar-17 10:36:21.995 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-17 10:36:21.995 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-17 10:36:22.001 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name kraken_pipeline:determine_bracken_length
Mar-17 10:36:22.002 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-17 10:36:22.002 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-17 10:36:22.008 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name kraken_pipeline:kraken_server
Mar-17 10:36:22.009 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-17 10:36:22.009 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-17 10:36:22.011 [main] INFO nextflow.Nextflow -
Mar-17 10:36:22.011 [main] INFO nextflow.Nextflow - Input directory assumed to be containing one or more directories containing fastq files.
Mar-17 10:36:22.045 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name kraken_pipeline:kraken2_client
Mar-17 10:36:22.046 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-17 10:36:22.046 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-17 10:36:22.065 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name kraken_pipeline:progressive_stats
Mar-17 10:36:22.066 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-17 10:36:22.066 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-17 10:36:22.086 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name kraken_pipeline:progressive_kraken_reports
Mar-17 10:36:22.086 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-17 10:36:22.086 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-17 10:36:22.108 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name kraken_pipeline:progressive_bracken
Mar-17 10:36:22.108 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-17 10:36:22.109 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-17 10:36:22.111 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name kraken_pipeline:getVersions
Mar-17 10:36:22.111 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-17 10:36:22.111 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-17 10:36:22.118 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name kraken_pipeline:getParams
Mar-17 10:36:22.119 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-17 10:36:22.119 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-17 10:36:22.126 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name kraken_pipeline:makeReport
Mar-17 10:36:22.126 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-17 10:36:22.126 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-17 10:36:22.131 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name kraken_pipeline:output
Mar-17 10:36:22.131 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-17 10:36:22.131 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-17 10:36:22.136 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name kraken_pipeline:stop_kraken_server
Mar-17 10:36:22.136 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-17 10:36:22.136 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-17 10:36:22.141 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: combineFilterFastq, unpackDatabase, kraken_server, kraken_pipeline:unpackTaxonomy, kraken_pipeline:stop_kraken_server, kraken_pipeline:kraken2_client, progressive_stats, kraken_pipeline:progressive_stats, output, stop_kraken_server, kraken_pipeline:makeReport, makeReport, minimap, getVersions, kraken_pipeline:progressive_kraken_reports, unpackTaxonomy, extractMinimap2Reads, rebatchFastq, kraken_pipeline:getParams, kraken_pipeline:unpackDatabase, kraken2_client, kraken_pipeline:getVersions, isolateSingleFile, progressive_bracken, determine_bracken_length, progressive_kraken_reports, kraken_pipeline:determine_bracken_length, checkSampleSheet, stopCondition, kraken_pipeline:kraken_server, getParams, kraken_pipeline:progressive_bracken, kraken_pipeline:output
Mar-17 10:36:22.141 [main] DEBUG nextflow.Session - Ignite dataflow network (22)
Mar-17 10:36:22.141 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > kraken_pipeline:unpackTaxonomy
Mar-17 10:36:22.141 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > kraken_pipeline:unpackDatabase
Mar-17 10:36:22.141 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > kraken_pipeline:determine_bracken_length
Mar-17 10:36:22.141 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > kraken_pipeline:kraken_server
Mar-17 10:36:22.149 [PathVisitor-1] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: /home/abiplantlab/epi2melabs/demo/epi2me-labs/wf-metagenomics/wf-metagenomics-demo/test_data/; pattern: **/*f*q*; options: [:]
Mar-17 10:36:22.151 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > kraken_pipeline:kraken2_client
Mar-17 10:36:22.152 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > kraken_pipeline:progressive_stats
Mar-17 10:36:22.153 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > kraken_pipeline:progressive_kraken_reports
Mar-17 10:36:22.153 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > kraken_pipeline:progressive_bracken
Mar-17 10:36:22.153 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > kraken_pipeline:getVersions
Mar-17 10:36:22.153 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > kraken_pipeline:getParams
Mar-17 10:36:22.153 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > kraken_pipeline:makeReport
Mar-17 10:36:22.154 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > kraken_pipeline:output
Mar-17 10:36:22.154 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > kraken_pipeline:stop_kraken_server
Mar-17 10:36:22.154 [main] DEBUG nextflow.script.ScriptRunner - > Await termination
Mar-17 10:36:22.154 [main] DEBUG nextflow.Session - Session await
Mar-17 10:36:22.339 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-17 10:36:22.341 [Task submitter] INFO nextflow.Session - [c1/52af41] Submitted process > kraken_pipeline:kraken2_client (8)
Mar-17 10:36:22.371 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-17 10:36:22.371 [Task submitter] INFO nextflow.Session - [4a/354869] Submitted process > kraken_pipeline:getVersions
Mar-17 10:36:22.381 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-17 10:36:22.381 [Task submitter] INFO nextflow.Session - [7b/a2f1a9] Submitted process > kraken_pipeline:getParams
Mar-17 10:36:22.652 [Actor Thread 9] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=4; maxSize=4; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Mar-17 10:36:22.781 [FileTransfer-1] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip at /home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/work/stage/eb/fadd9cb6329e266d718a18310c9408/taxdmp_2023-01-01.zip
Mar-17 10:36:23.392 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-17 10:36:23.392 [Task submitter] INFO nextflow.Session - [9d/01f361] Submitted process > kraken_pipeline:unpackTaxonomy
Mar-17 10:36:23.928 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 3; name: kraken_pipeline:getParams; status: COMPLETED; exit: 1; error: -; workDir: /home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/work/7b/a2f1a9d35dd775f165e7359897617d]
Mar-17 10:36:23.959 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'kraken_pipeline:getParams'
Caused by:
Process `kraken_pipeline:getParams` terminated with an error exit status (1)
Command executed:
# Output nextflow params object to JSON
echo '{
"help": false,
"version": false,
"fastq": "/home/abiplantlab/epi2melabs/demo/epi2me-labs/wf-metagenomics/wf-metagenomics-demo/test_data",
"sample": null,
"sample_sheet": null,
"max_len": null,
"min_len": 0,
"taxonomy": null,
"classifier": "kraken2",
"reference": null,
"ref2taxid": null,
"minimap2filter": null,
"minimap2exclude": false,
"split_prefix": false,
"database": null,
"bracken_dist": null,
"bracken_length": null,
"bracken_level": "S",
"out_dir": "/home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/output",
"disable_ping": false,
"threads": 2,
"aws_image_prefix": null,
"aws_queue": null,
"batch_size": 0,
"watch_path": false,
"store_dir": "store_dir",
"read_limit": null,
"port": 8080,
"database_set": "ncbi_16s_18s",
"database_sets": {
"ncbi_16s_18s": {
"reference": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna",
"refindex": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna.fai",
"database": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz",
"kmer_dist": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib",
"ref2taxid": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ref2taxid.targloci.tsv",
"taxonomy": "https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip"
},
"ncbi_16s_18s_28s_ITS": {
"reference": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna",
"refindex": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna.fai",
"database": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS_kraken2.tar.gz",
"kmer_dist": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/database1000mers.kmer_distrib",
"ref2taxid": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ref2taxid.ncbi_16s_18s_28s_ITS.tsv",
"taxonomy": "https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip"
},
"PlusPF-8": {
"database": "https://genome-idx.s3.amazonaws.com/kraken/k2_pluspf_08gb_20221209.tar.gz",
"taxonomy": "https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2022-09-01.zip"
}
},
"process_label": "wfmetagenomics",
"monochrome_logs": false,
"validate_params": true,
"show_hidden_params": false,
"analyse_unclassified": false,
"schema_ignore_params": "show_hidden_params,validate_params,monochrome_logs,aws_queue,aws_image_prefix,pangolin_version,wf,process_label",
"wf": {
"example_cmd": [
"--fastq test_data/barcode01/reads.fastq.gz"
],
"agent": "epi2melabs/4.1.3",
"container_sha": "sha1e115f37e792823e57976e905c62e631d4633206"
}
}' > params.json
Command exit status:
1
Command output:
(empty)
Command error:
touch: cannot touch '.command.trace': Permission denied
Work dir:
/home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/work/7b/a2f1a9d35dd775f165e7359897617d
Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
Mar-17 10:36:23.980 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process `kraken_pipeline:getParams` terminated with an error exit status (1)
Mar-17 10:36:24.004 [Task monitor] DEBUG nextflow.Session - The following nodes are still active:
[process] kraken_pipeline:unpackDatabase
status=ACTIVE
port 0: (value) bound ; channel: database
port 1: (value) bound ; channel: kmer_distribution
port 2: (cntrl) - ; channel: $
[process] kraken_pipeline:determine_bracken_length
status=ACTIVE
port 0: (value) OPEN ; channel: database
port 1: (cntrl) - ; channel: $
[process] kraken_pipeline:kraken_server
status=ACTIVE
port 0: (value) OPEN ; channel: database
port 1: (cntrl) - ; channel: $
[process] kraken_pipeline:progressive_stats
status=ACTIVE
port 0: (queue) OPEN ; channel: fastcat_stats
port 1: (cntrl) - ; channel: $
[process] kraken_pipeline:progressive_kraken_reports
status=ACTIVE
port 0: (queue) OPEN ; channel: kreport
port 1: (queue) OPEN ; channel: sample_ids
port 2: (value) OPEN ; channel: taxonomy
port 3: (cntrl) - ; channel: $
[process] kraken_pipeline:progressive_bracken
status=ACTIVE
port 0: (queue) OPEN ; channel: inputs
port 1: (queue) OPEN ; channel: sample_ids
port 2: (value) OPEN ; channel: -
port 3: (cntrl) - ; channel: $
[process] kraken_pipeline:makeReport
status=ACTIVE
port 0: (queue) OPEN ; channel: lineages
port 1: (queue) OPEN ; channel: -
port 2: (cntrl) - ; channel: $
[process] kraken_pipeline:output
status=ACTIVE
port 0: (queue) OPEN ; channel: fname
port 1: (cntrl) - ; channel: $
[process] kraken_pipeline:stop_kraken_server
status=ACTIVE
port 0: (value) OPEN ; channel: stop
port 1: (cntrl) - ; channel: $
Mar-17 10:36:24.149 [main] DEBUG nextflow.Session - Session await > all process finished
Mar-17 10:36:24.150 [main] DEBUG nextflow.Session - Session await > all barriers passed
Mar-17 10:36:24.299 [main] WARN n.processor.TaskPollingMonitor - Killing running tasks (3)
Mar-17 10:36:24.309 [main] DEBUG nextflow.executor.LocalTaskHandler - Unable to kill kraken_pipeline:getVersions -- command: kill -TERM 148195; exit: 1
bash: line 0: kill: (148195) - No such process
Mar-17 10:36:24.318 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=8; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=3; succeedDuration=0ms; failedDuration=1.5s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=4; peakCpus=4; peakMemory=0; ]
Mar-17 10:36:24.318 [main] DEBUG nextflow.trace.TraceFileObserver - Flow completing -- flushing trace file
Mar-17 10:36:24.320 [main] DEBUG nextflow.trace.ReportObserver - Flow completing -- rendering html report
Mar-17 10:36:24.343 [main] DEBUG nextflow.trace.ReportObserver - Execution report summary data:
[{"cpuUsage":null,"process":"getParams","mem":null,"memUsage":null,"timeUsage":null,"vmem":null,"reads":null,"cpu":null,"time":{"mean":1488,"min":1488,"q1":1488,"q2":1488,"q3":1488,"max":1488,"minLabel":"kraken_pipeline:getParams","maxLabel":"kraken_pipeline:getParams","q1Label":"kraken_pipeline:getParams","q2Label":"kraken_pipeline:getParams","q3Label":"kraken_pipeline:getParams"},"writes":null},{"cpuUsage":null,"process":"kraken2_client","mem":null,"memUsage":null,"timeUsage":null,"vmem":null,"reads":null,"cpu":null,"time":null,"writes":null},{"cpuUsage":null,"process":"getVersions","mem":null,"memUsage":null,"timeUsage":null,"vmem":null,"reads":null,"cpu":null,"time":null,"writes":null},{"cpuUsage":null,"process":"unpackTaxonomy","mem":null,"memUsage":null,"timeUsage":null,"vmem":null,"reads":null,"cpu":null,"time":null,"writes":null}]
Mar-17 10:36:24.437 [FileTransfer-2] DEBUG nextflow.file.FilePorter - Local cache found for foreign file https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz at /home/abiplantlab/epi2melabs/instances/wf-metagenomics_30807338-7952-49c2-a7ce-28712da563b8/work/stage/da/1b7f80741b42bb7340b4bc109a2f75/ncbi_targeted_loci_kraken2.tar.gz
Mar-17 10:36:24.886 [main] DEBUG nextflow.trace.TimelineObserver - Flow completing -- rendering html timeline
Mar-17 10:36:24.981 [main] WARN nextflow.dag.GraphvizRenderer - To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.
Mar-17 10:36:25.036 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Mar-17 10:36:25.044 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye[Bug]:
What happened?
A bug happened!
Operating System
macOS
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
None
Workflow Version
2.2.1
Relevant log output
ERROR ~ Error executing process > 'kraken_pipeline:getVersions'
Caused by:
Process `kraken_pipeline:getVersions` terminated with an error exit status (127)
Command executed:
python -c "import pysam; print(f'pysam,{pysam.__version__}')" >> versions.txt
python -c "import pandas; print(f'pandas,{pandas.__version__}')" >> versions.txt
fastcat --version | sed 's/^/fastcat,/' >> versions.txt
minimap2 --version | sed 's/^/minimap2,/' >> versions.txt
samtools --version | head -n 1 | sed 's/ /,/' >> versions.txt
taxonkit version | sed 's/ /,/' >> versions.txt
kraken2 --version | head -n 1 | sed 's/ version /,/' >> versions.txt
Command exit status:
127
Command output:
(empty)
Command error:
.command.run: line 290: docker: command not found
Work dir:
/Users/luciaq/work/7f/0751fd2569b36cd84f086b3680937a
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
-- Check '.nextflow.log' file for details
WARN: Graphviz is required to render the execution DAG in the given format -- See http://www.graphviz.org for more info.[Bug]: `--threads` option triggers a bug
What happened?
I'm running the pipeline using Nextflow CLI. When I specify a number of threads other than the default using the --threads option, I notice kraken2_server is not able to start. But if I don't change the number of threads then kraken2_server starts without problems.
Until now, my workaround has been to start the server in a separate shell, but I just noticed that if I don't change the number of threads kraken2_server works just fine!
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
Docker
Workflow Version
2.0.8
Relevant log output
Connecting to server: localhost:8080.
Classifying sequence stream.
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...And i am not able to find any proper manual or tutorial for EPI2ME command line version using nextflow with docker?
nextflow run epi2me-labs/wf-metagenomics --fastq reads.fastq.gz -profile standard --out_dir test1
I am using this command with the test data available but there is nothing in the output?
And i am not able to find any proper manual or tutorial for EPI2ME command line version using nextflow with docker?
Minimap2 usage
What happened?
I am trying to use this workflow with --classifier minimap2 but I am unsure on how to correctly use this option. I try running:
nextflow run epi2me-labs/wf-metagenomics --fastq ../anonymous_reads_sample_0.fq.gz --classifier minimap2 --reference archaea.fna --out_dir test --threads 20In this test run I created a fasta file with complete archaeal genomes. I am using simulated reads so I know that in my fastq file I have some archaeal genomes present. I also tried running it with a smaller reference file with viruses.
Every time I run it, the workflow "completes successfully" but the output files are empty. I only get an empty wf-metagenomics-report.html file in the output directory. Also, it only runs for just a few minutes compared to the kraken2 workflow.
So my question is what is the right way to use this workflow with minimap2 and what would the command to execute look like?
Operating System
ubuntu 18.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
None
Workflow Version
v2.0.6
Relevant log output
--------------------------------------------------------------------------------
This is epi2me-labs/wf-metagenomics v2.0.6-ga67b931.
--------------------------------------------------------------------------------
Checking inputs.
Checking custom reference exists
Checking custom reference index exists
Checking fastq input.
Single file input detected.
executor > local (9)
[fb/c1a177] process > minimap_pipeline:unpackTaxonomy [100%] 1 of 1 ✔
[ca/8e1795] process > minimap_pipeline:combineFilterFastq (1) [100%] 1 of 1 ✔
[b4/eb5976] process > minimap_pipeline:minimap (1) [100%] 1 of 1 ✔
[d4/3341c0] process > minimap_pipeline:getVersions [100%] 1 of 1 ✔
[a5/5d690a] process > minimap_pipeline:getParams [100%] 1 of 1 ✔
[26/eb843b] process > minimap_pipeline:makeReport [100%] 1 of 1 ✔
[b8/4363d7] process > minimap_pipeline:output (3) [100%] 3 of 3 ✔
Completed at: 03-Jan-2023 09:51:44
Duration : 4m
CPU hours : 0.3
Succeeded : 9Process `pipeline:minimap2 (1)` terminated with an error exit status (1)
Hi,
I am currently trying a few data analysis options to detect viral contaminations in my sample. Therefore, I also wanted to test the metagenomics workflow, and also compare the minimap2 and the kraken2 option. When I run the minimap2 option, the run just stops at the minimap2 step. This is the overall ouput I receive:
nextflow run epi2me-labs/wf-metagenomics -r v1.1.4 --fastq all.fastq --out_dir nf_minimap_test/ --minimap2 --reference RefSeq_viral.fna --kraken2 FALSE --threads 20
N E X T F L O W ~ version 22.04.5
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [f3e7568d46]
Launching https://github.com/epi2me-labs/wf-metagenomics [determined_hugle] DSL2 - revision: a0bd9ca [v1.1.4]
Core Nextflow options
revision : v1.1.4
runName : determined_hugle
containerEngine: docker
launchDir : /home/hans-peter/Test_Area
workDir : /home/hans-peter/Test_Area/work
projectDir : /home/hans-peter/.nextflow/assets/epi2me-labs/wf-metagenomics
userName : hans-peter
profile : standard
configFiles : /home/hans-peter/.nextflow/assets/epi2me-labs/wf-metagenomics/nextflow.config
Core options
fastq : all.fastq
out_dir : nf_minimap_test/
sources : [ncbi_16s_18s:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ref2taxid.targloci.tsv, taxonomy:https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz], ncbi_16s_18s_28s_ITS:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ref2taxid.ncbi_16s_18s_28s_ITS.tsv, taxonomy:https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz], PlusPF-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspf_8gb_20210517.tar.gz, taxonomy:https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz]]
Minimap2 options
minimap2 : true
reference : RefSeq_viral.fna
Kraken2 options
kraken2 : false
Generic options
threads : 20
!! Only displaying parameters that differ from the pipeline defaults !!
If you use wf-metagenomics for your analysis please cite:
- The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
Checking inputs.
Checking custom reference exists
Checking custom reference index exists
Checking fastq input.
Single file input detected.
executor > local (6)
[52/9ff427] process > handleSingleFile (1) [100%] 1 of 1 ✔
[97/7a3645] process > pipeline:unpackTaxonomy [100%] 1 of 1 ✔
[a4/795cdf] process > pipeline:combineFilterFastq (1) [100%] 1 of 1 ✔
[dc/c50135] process > pipeline:minimap2 (1) [ 0%] 0 of 1
[47/eec279] process > pipeline:getVersions [100%] 1 of 1 ✔
[f0/73726d] process > pipeline:getParams [100%] 1 of 1 ✔
[- ] process > pipeline:makeReport -
[- ] process > output -
Error executing process > 'pipeline:minimap2 (1)'
Caused by:
Process pipeline:minimap2 (1) terminated with an error exit status (1)
Command executed:
executor > local (6)
[52/9ff427] process > handleSingleFile (1) [100%] 1 of 1 ✔
[97/7a3645] process > pipeline:unpackTaxonomy [100%] 1 of 1 ✔
[a4/795cdf] process > pipeline:combineFilterFastq (1) [100%] 1 of 1 ✔
[dc/c50135] process > pipeline:minimap2 (1) [100%] 1 of 1, failed: 1 ✘
[47/eec279] process > pipeline:getVersions [100%] 1 of 1 ✔
[f0/73726d] process > pipeline:getParams [100%] 1 of 1 ✔
[- ] process > pipeline:makeReport -
[- ] process > output [ 0%] 0 of 2
Error executing process > 'pipeline:minimap2 (1)'
Caused by:
Process pipeline:minimap2 (1) terminated with an error exit status (1)
Command executed:
minimap2 -t 20 -ax map-ont RefSeq_viral.fna all.fastq | samtools view -h -F 2304 - | format_minimap2.py - -o all.minimap2.assignments.tsv -r ref2taxid.targloci.tsv | samtools sort -o all.bam -
samtools index all.bam
awk -F '\t' '{print $3}' all.minimap2.assignments.tsv > taxids.tmp
taxonkit --data-dir taxonomy_dir lineage -R taxids.tmp | aggregate_lineages.py -p all.minimap2
Command exit status:
1
Command output:
(empty)
Command error:
[M::mm_idx_gen::18.2250.83] collected minimizers
[M::mm_idx_gen::43.7980.91] sorted minimizers
[M::main::43.8070.91] loaded/built the index for 14813 target sequence(s)
[M::mm_mapopt_update::44.4230.91] mid_occ = 92
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 14813
[M::mm_idx_stat::44.723*0.91] distinct minimizers: 40131811 (62.74% are singletons); average occurrences: 2.185; average spacing: 5.355; total length: 469589088
Traceback (most recent call last):
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3621, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 163, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'NC_018464.1'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/hans-peter/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/format_minimap2.py", line 82, in
execute(sys.argv[1:])
File "/home/hans-peter/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/format_minimap2.py", line 74, in execute
main(
File "/home/hans-peter/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/format_minimap2.py", line 30, in main
taxid = ref2taxid_df.at[aln.reference_name, 'taxid']
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/indexing.py", line 2270, in getitem
return super().getitem(key)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/indexing.py", line 2221, in getitem
return self.obj._get_value(*key, takeable=self._takeable)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/frame.py", line 3622, in _get_value
row = self.index.get_loc(index)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3623, in get_loc
raise KeyError(key) from err
KeyError: 'NC_018464.1'
Work dir:
/home/hans-peter/Test_Area/work/dc/c50135909fb31f19693d617d7f1bdb
Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line
help would be greatly appreciated :)
thanks and best regards,
Hans-Peter
[Bug]: kraken_pipeline:kraken_server terminated
What happened?
Everytime I run wf_metagenomics through EPI2ME Labs application, it always outputs that "Process kraken_pipeline:kraken_server terminated with an error exit status (71) -- Error is ignored". The actual run never finishes.
I am using Kraken2 with the PlusPFP-8 database on a single .fastq file containing multiple sequences. The rest of the settings are default.
Any help/tips as to why this is happening is much appreciated!
Operating System
macOS
Workflow Execution
EPI2ME Labs desktop application
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
None
Workflow Version
wf-metagenomics v2.2.1
Relevant log output
Checking inputs.
Checking fastq input.
[98/fd892e] Submitted process > fastcat (1)
[fa/251be0] Submitted process > kraken_pipeline:getVersions
[c6/a69df6] Submitted process > kraken_pipeline:getParams
[11/019f91] Submitted process > kraken_pipeline:output (1)
[8b/e808e0] Submitted process > kraken_pipeline:kraken2_client (1)
Staging foreign file: https://genome-idx.s3.amazonaws.com/kraken/k2_pluspfp_08gb_20230314.tar.gz
Staging foreign file: https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/new_taxdump_2023-03-01.zip
[6f/4335d4] Submitted process > kraken_pipeline:output (2)
[f2/685c69] Submitted process > kraken_pipeline:unpackTaxonomy
[1b/fe2ace] Submitted process > kraken_pipeline:unpackDatabase
[a5/65d6ac] Submitted process > kraken_pipeline:determine_bracken_length
[4f/37bf99] Submitted process > kraken_pipeline:kraken_server
[4f/37bf99] NOTE: Process `kraken_pipeline:kraken_server` terminated with an error exit status (71) -- Error is ignored[Bug]: Numpy version
What happened?
Numpy is complaining about the c components. It might be a version issue. It's the first time I encounter this error. I don't know if to do with how the dependencies are managed in the container. Error message and traceback below:
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
Docker
Workflow Version
2.0.8
Relevant log output
Error executing process > 'kraken_pipeline:kraken2_client (27)'
Caused by:
Process `kraken_pipeline:kraken2_client (27)` terminated with an error exit status (1)
Command executed:
fastcat -a "0" -s "CB0981_35_cycle_ITS" -r "CB0981_35_cycle_ITS.27.stats" "CB0981_35_cycle_ITS.fastq" > filtered.fastq
fastcat_histogram.py --sample_id "CB0981_35_cycle_ITS" "CB0981_35_cycle_ITS.27.stats" "CB0981_35_cycle_ITS.27.json"
kraken2_client --port 8080 --report report.txt --sequence filtered.fastq > "CB0981_35_cycle_ITS.kraken2.assignments.tsv"
tail -n +1 report.txt > "CB0981_35_cycle_ITS.kraken2.report.txt"
Command exit status:
1
Command output:
(empty)
Command error:
Traceback (most recent call last):
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numpy/core/__init__.py", line 23, in <module>
from . import multiarray
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numpy/core/multiarray.py", line 10, in <module>
from . import overrides
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numpy/core/overrides.py", line 6, in <module>
from numpy.core._multiarray_umath import (
ImportError: PyCapsule_Import could not import module "datetime"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/concertbio/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/fastcat_histogram.py", line 8, in <module>
import numpy as np
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numpy/__init__.py", line 140, in <module>
from . import core
File "/home/epi2melabs/conda/lib/python3.8/site-packages/numpy/core/__init__.py", line 49, in <module>
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.8 from "/home/epi2melabs/conda/bin/python"
* The NumPy version is: "1.23.5"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: PyCapsule_Import could not import module "datetime"
Work dir:
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`[Bug]: --batch_size > 1 do not work
Running in kraken2 mode with --batch_size > 1 is failling. I figured out that the error was due to an antire array with all fastq names being given to fastcat -x option (kraken_pipeline.nf line 81) instead of a multi argument, so I did this to the process combineFilterFastq:
process combineFilterFastq {
label "wfmetagenomics"
forks = params.threads.intdiv(6)*5
maxForks forks <= 0 ? 1 : forks
cpus 1
input:
tuple val(sample_id), path(directory)
path database
output:
tuple(
val(sample_id),
path("*.fastq.gz"),
emit: filtered)
tuple(
val(sample_id),
path("*.json"),
emit: stats)
shell:
"""
A=\$(echo "${directory}" | sed 's/\\[//g' | sed 's/\\]//g' | sed 's/,//g')
fastcat \
-a "${params.min_len}" \
-b "${params.max_len}" \
-q 10 \
-s "${sample_id}" \
-r "${sample_id}.${task.index}.stats" \
-x \$A | bgzip > "${sample_id}.${task.index}.fastq.gz"
fastcat_histogram.py --file "${sample_id}.${task.index}.stats" --sample_id "$sample_id"
"""
}
$A variable was created to remove the characters "[", "]" and "," and was given to -x.
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
ontresearch/wf-metagenomics
Workflow Execution - Execution Profile
Docker
Workflow Version
v2.0.1
Relevant log output
the error message was from fastcat (lost the logs already) and was saying that the input fastq name was too long and printed an array of length = batch_size of fastq names.[Bug]: error while aggregating kraken report with bracken
What happened?
Hello,
I am encountering a bug with the latest versions of the workflow when aggregating kraken reports with bracken. Something is wrong with the parsing of the kraken report or on the lineage, I can't identify precisely the source of the problem. Maybe one of you can help me.
I specify that I don't have any problem with version v1.1.4
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - Execution Profile
Docker
Workflow Version
v.2.0.3
Relevant log output
nextflow run epi2me-labs/wf-metagenomics --max_len 2000 --min_len 1000 --threads 16 --bracken_level "G" -r v2.0.3 --fastq fastq_pass/ --out_dir output_last_v2.0.3
Error executing process > 'kraken_pipeline:bracken (24)'
Caused by:
Process `kraken_pipeline:bracken (24)` terminated with an error exit status (1)
Command executed:
run_bracken.py "database_dir" "reports.24/barcode02.kreport.txt" "1000" "G" "barcode02.bracken_report.txt"
mv "reports.24/barcode02.kreport_bracken_species.txt" . || echo "no bracken report"
awk -F '( ' -v OFS=' ' '{ print $2,$6 }' "barcode02.bracken_report.txt" | awk -F ' ' -v OFS=' ' 'NR!=1 {print}' | tee taxacounts.txt | awk -F '' -v OFS='"/home/' '{ print $1 }' > taxa.txt
taxonkit --data-dir taxonomy_dir lineage -R taxa.txt > lineages.txt
aggregate_lineages_bracken.py -i "lineages.txt" -b "taxacounts.txt" -u "reports.24/barcode02.kreport.txt" -p "barcode02.kraken2"
file1=`cat *.json`
echo "{"'"barcode02"'": "$file1"}" >> "barcode02.24.json"
cp "barcode02.24.json" "reports.24/barcode02.json"
Command exit status:
1
Command output:
b' >> Checking for Valid Options...\n >> Running Bracken \n >> python src/est_abundance.py -i reports.24/barcode02.kreport.txt -o barcode02.bracken_report.txt -k database_dir/database1000mers.kmer_distrib -l G -t 0\nPROGRAM START TIME: 11-22-2022 18:32:43\nBRACKEN SUMMARY (Kraken report: reports.24/barcode02.kreport.txt)\n >>> Threshold: 0 \n >>> Number of genuses in sample: 453 \n\t >> Number of genuses with reads > threshold: 453 \n\t >> Number of genuses with reads < threshold: 0 \n >>> Total reads in sample: 22330\n\t >> Total reads kept at genuses level (reads > threshold): 17909\n\t >> Total reads discarded (genuses reads < threshold): 0\n\t >> Reads distributed: 1407\n\t >> Reads not distributed (eg. no genuses above threshold): 3014\n\t >> Unclassified reads: 0\nBRACKEN OUTPUT PRODUCED: barcode02.bracken_report.txt\nPROGRAM END TIME: 11-22-2022 18:32:43\n Bracken complete.\n'no bracken report
Command error:
b'>> Checking report file: reports.24/barcode02.kreport.txt\n'mv: cannot stat 'reports.24/barcode02.kreport_bracken_species.txt': No such file or directory
18:32:50.221 [WARN] taxid 2787116 was deleted
Traceback (most recent call last):
File "/home/administrateur/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/aggregate_lineages_bracken.py", line 41, in update_or_create_count
tax_id, lineage, ranks = entry.rstrip().split('\t')
ValueError: not enough values to unpack (expected 3, got 1)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/administrateur/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/aggregate_lineages_bracken.py", line 165, in <module>
execute()
File "/home/administrateur/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/aggregate_lineages_bracken.py", line 156, in execute
main(
File "/home/administrateur/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/aggregate_lineages_bracken.py", line 98, in main
entries = update_or_create_count(line, entries, bracken_counts)
File "/home/administrateur/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/aggregate_lineages_bracken.py", line 46, in update_or_create_count
sys.stderr('Error: unknown lineage {}'.format(entry))
TypeError: '_io.TextIOWrapper' object is not callable
Work dir:
/home/administrateur/dev/test_wf-metagenomics/work/f7/f306c88c184fafff74910a1f00553a
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.
## The first lines of the kraken report look fine
head barcode02.kreport.txt
0.0000 0 0 U 0 unclassified
100.0000 22330 0 R 1 root
100.0000 22330 0 R1 131567 cellular organisms
100.0000 22330 34 D 2 Bacteria
66.5741 14866 60 D1 1783272 Terrabacteria group
63.9006 14269 219 P 1239 Firmicutes
58.0430 12961 16 C 186801 Clostridia
57.7564 12897 597 O 186802 Eubacteriales
30.8106 6880 1095 F 186803 Lachnospiraceae
5.7456 1283 110 G 572511 Blautiaflag for sample_sheet does not work.
What happened?
--Sample_sheet flag does not work. I get the workflow to work with a directory containing barcode directories but not when using a sample sheet.
I have tried this on ubuntu and linux and I get the same error - "Missing process or function with name 'get_sample_sheet'"
edit: What format on the sample_sheet is expected from the pipeline? csv or tsv? Name of headers?
There also seem to be a problem with the --threads flag. When you try to use 8 or more than 8, the program crashes.
One last question: Which nextflow version do you reccomend for wf-metagenomics v.2.0.6?
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
nextflow run epi2me-labs/wf-metagenomics -r v2.0.6 --fastq test_data/ -profile singularity --sample_sheet sample_sheet.txt
Workflow Execution - CLI Execution Profile
Singularity
Workflow Version
v2.0.6
Relevant log output
N E X T F L O W ~ version 22.04.0
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [a3a9d27cc2]
Launching `https://github.com/epi2me-labs/wf-metagenomics` [distracted_kalman] DSL2 - revision: f5d1e16131 [v2.0.6]
WARN: NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASE
|||||||||| _____ ____ ___ ____ __ __ _____ _ _
|||||||||| | ____| _ \_ _|___ \| \/ | ____| | | __ _| |__ ___
||||| | _| | |_) | | __) | |\/| | _| _____| |/ _` | '_ \/ __|
||||| | |___| __/| | / __/| | | | |__|_____| | (_| | |_) \__ \
|||||||||| |_____|_| |___|_____|_| |_|_____| |_|\__,_|_.__/|___/
|||||||||| wf-metagenomics v2.0.6-gf5d1e16
--------------------------------------------------------------------------------
Core Nextflow options
revision : v2.0.6
runName : distracted_kalman
containerEngine: singularity
launchDir : /aux/db/workdir_fwa010/wf-metagenomics
workDir : /aux/db/workdir_fwa010/wf-metagenomics/work
projectDir : /home/orebroll.se/fwa010/.nextflow/assets/epi2me-labs/wf-metagenomics
userName : fwa010
profile : singularity
configFiles : /home/orebroll.se/fwa010/.nextflow/assets/epi2me-labs/wf-metagenomics/nextflow.config, /aux/db/workdir_fwa010/wf-metagenomics/nextflow.config
Input Options
fastq : test_data/
Sample Options
sample_sheet : sample_sheet.csv
Reference Options
database_sets : [ncbi_16s_18s:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ref2taxid.targloci.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz], ncbi_16s_18s_28s_ITS:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ref2taxid.ncbi_16s_18s_28s_ITS.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz], PlusPF-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspf_8gb_20210517.tar.gz, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz]]
Other parameters
process_label : wfmetagenomics
!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-metagenomics for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
--------------------------------------------------------------------------------
This is epi2me-labs/wf-metagenomics v2.0.6-gf5d1e16.
--------------------------------------------------------------------------------
Checking inputs.
[- ] process > kraken_pipeline:unpackTaxonomy -
[- ] process > kraken_pipeline:unpackDatabase -
[- ] process > kraken_pipeline:determine_bracken_length -
[- ] process > kraken_pipeline:kraken_server -
Input directory assumed to be containing one or more directories containing fastq files.
Missing process or function with name 'get_sample_sheet'
-- Check script '/home/orebroll.se/fwa010/.nextflow/assets/epi2me-labs/wf-metagenomics/main.nf' at line: 140 or see '.nextflow.log' file for more detailsThis workflow can work for sample.fastq.gz using zymo database, however, Not work in hpvc database.
This workflow can work for sample.fastq.gz using zymo database, however, Not work in hpvc database. I wanna know which cause this error. Thanks!
It can work as follows:
OUTPUT=output
nextflow run main.nf \
-c my_config.cfg \
-w ${OUTPUT}/workspace \
-profile standard \
--fastq test_data/sample.fastq.gz \
--db_path test_data/db_store/zymo \
--db_prefix zymo \
--out_dir ${OUTPUT} \
--threads 40 \
--wfversion latestIt can NOT work as follows:
OUTPUT=output
nextflow run main.nf \
-c my_config.cfg \
-w ${OUTPUT}/workspace \
-profile standard \
--fastq test_data/sample.fastq.gz \
--db_path test_data/db_store \
--db_prefix hpvc \
--out_dir ${OUTPUT} \
--threads 40 \
--wfversion latestError info
Error executing process > 'pipeline:generateMaster (1)'
Caused by:
Process `pipeline:generateMaster (1)` terminated with an error exit status (1)
Command executed:
generate_master_table.py analysis/read_classifications.tsv seqs.txt analysis --split "fungi:phylum:4751 bacteria:phylum:2 viruses:phylum:10239 else:superkingdom:" --human
generate_report.py analysis/read_classification_master.tsv seqs.txt
date
Command exit status:
1
Command output:
(empty)
Command error:
/home/epi2melabs/conda/lib/python3.8/site-packages/ncbitaxonomy/ncbi_taxonomy/ncbiquery.py:233: UserWarning: taxid 2588707 was translated into 2487422
warnings.warn("taxid %s was translated into %s" %(taxid, merged_conversion[taxid]))
Traceback (most recent call last):
File "/root/project/nanopore/wf-metagenomics/bin/generate_master_table.py", line 304, in <module>
main()
File "/root/project/nanopore/wf-metagenomics/bin/generate_master_table.py", line 277, in main
assignments = by_rule(tax_groups, rules=split_by_rule)
File "/root/project/nanopore/wf-metagenomics/bin/generate_master_table.py", line 211, in by_rule
for rule in _generate_rules(rules):
File "/root/project/nanopore/wf-metagenomics/bin/gene
My enviroment infomation
Python version 3.8.10
nextflow version 21.04.0
project name: epi2me-labs/wf-metagenomics repository : https://github.com/epi2me-labs/wf-metagenomics
local path : /root/.nextflow/assets/epi2me-labs/wf-metagenomics
main script : main.nf
revisions :
* master (default)
prerelease
v0.0.1 [t]
v0.0.2 [t]
v0.1.0 [t]
v0.2.0 [t]
Permission issue
What happened?
I'm trying to run the workflow in command line, but I have an issue with docker permissions. I ran the following command:
nextflow run epi2me-labs/wf-metagenomics --fastq test_data --kraken2
Operating System
ubuntu 18.04
Workflow Execution
Command line
Workflow Execution - Execution Profile
Docker
Workflow Version
v2.0.1
Relevant log output
Error executing process > 'kraken_pipeline:getVersions'
Caused by:
Process `kraken_pipeline:getVersions` terminated with an error exit status (1)
Command executed:
python -c "import pysam; print(f'pysam,{pysam.__version__}')" >> versions.txt
python -c "import pandas; print(f'pandas,{pandas.__version__}')" >> versions.txt
fastcat --version | sed 's/^/fastcat,/' >> versions.txt
minimap2 --version | sed 's/^/minimap2,/' >> versions.txt
samtools --version | head -n 1 | sed 's/ /,/' >> versions.txt
taxonkit version | sed 's/ /,/' >> versions.txt
kraken2 --version | head -n 1 | sed 's/ version /,/' >> versions.txt
Command exit status:
1
Command output:
(empty)
Command error:
touch: cannot touch '.command.trace': Permission denied
Work dir:
/home/gaudillf/wf-metagenomics/work/79/133b83bd3e5f4547948f1a8739a0f3
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
Apparently the error is similar to what is mentioned here: assemblerflow/flowcraft#142
I tried the suggested fix, but it didn't work. Docker itself runs fine in command line (without sudo), so the permissions should be working...
It can work
[Bug]: Lineage plot shows all unclassified?
What happened?
hello!
Thank you for this amazing program! <3
When running the program everything seems to be running and executing properly. Although when I look at the output the Lineage Plot shows all unclassified and doesn't classify any of the reads. When running the test_bank dataset I am getting classified outputs for the lineage plot. Where am I going wrong?
I have tried processing a bacillus subtillis, legionella pneumophila, and a tumor dataset and all have resulted in all unclassified for the lineage plot.
I have run this workflow on both the command line as well as the EPI2ME Labs desktop application.
I have attached the log for the run below and a screenshot of what exactly I am talking about.
THANK YOU <3
Operating System
ubuntu 20.04
Workflow Execution
EPI2ME Labs desktop application
Workflow Execution - EPI2ME Labs Versions
EPI2ME Labs 3.1.5 Labs Environment v1.2.5
Workflow Execution - Execution Profile
Docker
Workflow Version
v1.1.4
Relevant log output
Oct-10 18:13:33.679 [main] DEBUG nextflow.cli.Launcher - $> nextflow -log /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/nextflow.log run epi2me-labs/wf-metagenomics -params-file /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/params.json -w /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work -ansi-log false -r v1.1.4
Oct-10 18:13:33.716 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 22.04.0
Oct-10 18:13:34.372 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/matteo/.nextflow/assets/epi2me-labs/wf-metagenomics/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-metagenomics.git
Oct-10 18:13:34.383 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/matteo/.nextflow/assets/epi2me-labs/wf-metagenomics/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-metagenomics.git
Oct-10 18:13:35.420 [main] INFO nextflow.scm.AssetManager - NOTE: Your local project version looks outdated - a different revision is available in the remote repository [f3e7568d46]
Oct-10 18:13:35.438 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /home/matteo/.nextflow/assets/epi2me-labs/wf-metagenomics/nextflow.config
Oct-10 18:13:35.439 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/matteo/.nextflow/assets/epi2me-labs/wf-metagenomics/nextflow.config
Oct-10 18:13:35.453 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Oct-10 18:13:35.580 [main] INFO nextflow.cli.CmdRun - Launching `https://github.com/epi2me-labs/wf-metagenomics` [adoring_visvesvaraya] DSL2 - revision: a0bd9ca590 [v1.1.4]
Oct-10 18:13:35.591 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; plugins-dir=/home/matteo/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Oct-10 18:13:35.592 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Oct-10 18:13:35.599 [main] INFO org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Oct-10 18:13:35.600 [main] INFO org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Oct-10 18:13:35.603 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Oct-10 18:13:35.609 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Oct-10 18:13:35.649 [main] DEBUG nextflow.Session - Session uuid: da84e578-bd20-4dc1-bcd7-2a6fac1c14c2
Oct-10 18:13:35.649 [main] DEBUG nextflow.Session - Run name: adoring_visvesvaraya
Oct-10 18:13:35.650 [main] DEBUG nextflow.Session - Executor pool size: 8
Oct-10 18:13:35.669 [main] DEBUG nextflow.cli.CmdRun -
Version: 22.04.0 build 5697
Created: 23-04-2022 18:00 UTC (11:00 PDT)
System: Linux 5.15.0-48-generic
Runtime: Groovy 3.0.10 on OpenJDK 64-Bit Server VM 11.0.16+8-post-Ubuntu-0ubuntu122.04
Encoding: UTF-8 (UTF-8)
Process: 34331@matteo-Thelio [127.0.1.1]
CPUs: 8 - Mem: 31.3 GB (1.3 GB) - Swap: 2 GB (2 GB)
Oct-10 18:13:35.682 [main] DEBUG nextflow.Session - Work-dir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work [ext2/ext3]
Oct-10 18:13:35.729 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[GoogleLifeSciencesExecutor, AwsBatchExecutor]
Oct-10 18:13:35.749 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Oct-10 18:13:35.762 [main] DEBUG nextflow.Session - Observer factory: TowerFactory
Oct-10 18:13:35.800 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Oct-10 18:13:35.807 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 9; maxThreads: 1000
Oct-10 18:13:35.857 [main] DEBUG nextflow.Session - Session start invoked
Oct-10 18:13:35.860 [main] DEBUG nextflow.trace.TraceFileObserver - Flow starting -- trace file: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/output/execution/trace.txt
Oct-10 18:13:35.864 [main] DEBUG nextflow.Session - Using default localLib path: /home/matteo/.nextflow/assets/epi2me-labs/wf-metagenomics/lib
Oct-10 18:13:35.866 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/matteo/.nextflow/assets/epi2me-labs/wf-metagenomics/lib
Oct-10 18:13:35.866 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/matteo/.nextflow/assets/epi2me-labs/wf-metagenomics/lib/nfcore_external_java_deps.jar
Oct-10 18:13:37.173 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Oct-10 18:13:37.501 [main] WARN nextflow.Nextflow - Found unexpected parameters:
* --wf: [agent:labslauncher/3.1.5]
Oct-10 18:13:37.501 [main] INFO nextflow.Nextflow - - �[2mIgnore this warning: params.schema_ignore_params = "wf" �[0m
Oct-10 18:13:37.508 [main] INFO nextflow.Nextflow - �[1mCore Nextflow options�[0m
�[0;34mrevision : �[0;32mv1.1.4�[0m
�[0;34mrunName : �[0;32madoring_visvesvaraya�[0m
�[0;34mcontainerEngine: �[0;32mdocker�[0m
�[0;34mlaunchDir : �[0;32m/home/matteo/epi2melabs-data/nextflow�[0m
�[0;34mworkDir : �[0;32m/home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work�[0m
�[0;34mprojectDir : �[0;32m/home/matteo/.nextflow/assets/epi2me-labs/wf-metagenomics�[0m
�[0;34muserName : �[0;32mmatteo�[0m
�[0;34mprofile : �[0;32mstandard�[0m
�[0;34mconfigFiles : �[0;32m/home/matteo/.nextflow/assets/epi2me-labs/wf-metagenomics/nextflow.config�[0m
�[1mCore options�[0m
�[0;34mfastq : �[0;32m/home/matteo/datasets/bsub/input_reads.fastq.gz�[0m
�[0;34mout_dir : �[0;32m/home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/output�[0m
�[0;34msources : �[0;32m[ncbi_16s_18s:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ref2taxid.targloci.tsv, taxonomy:https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz], ncbi_16s_18s_28s_ITS:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ref2taxid.ncbi_16s_18s_28s_ITS.tsv, taxonomy:https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz], PlusPF-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspf_8gb_20210517.tar.gz, taxonomy:https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz]]�[0m
!! Only displaying parameters that differ from the pipeline defaults !!
-�[2m----------------------------------------------------�[0m-
If you use wf-metagenomics for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
Oct-10 18:13:37.518 [main] INFO nextflow.Nextflow - Checking inputs.
Oct-10 18:13:37.590 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name handleSingleFile
Oct-10 18:13:37.594 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Oct-10 18:13:37.594 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Oct-10 18:13:37.598 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Oct-10 18:13:37.602 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=4; memory=8 GB; capacity=8; pollInterval=100ms; dumpInterval=5m
Oct-10 18:13:37.687 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name pipeline:unpackTaxonomy
Oct-10 18:13:37.688 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Oct-10 18:13:37.688 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Oct-10 18:13:37.691 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name pipeline:unpackDatabase
Oct-10 18:13:37.691 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Oct-10 18:13:37.691 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Oct-10 18:13:37.707 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name pipeline:combineFilterFastq
Oct-10 18:13:37.708 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Oct-10 18:13:37.708 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Oct-10 18:13:37.715 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name pipeline:kraken2
Oct-10 18:13:37.716 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Oct-10 18:13:37.716 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Oct-10 18:13:37.729 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name pipeline:bracken
Oct-10 18:13:37.730 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Oct-10 18:13:37.730 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Oct-10 18:13:37.733 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name pipeline:getVersions
Oct-10 18:13:37.733 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Oct-10 18:13:37.733 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Oct-10 18:13:37.735 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name pipeline:getParams
Oct-10 18:13:37.735 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Oct-10 18:13:37.735 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Oct-10 18:13:37.748 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name pipeline:makeReport
Oct-10 18:13:37.748 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Oct-10 18:13:37.748 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Oct-10 18:13:37.759 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:wfmetagenomics` matches labels `wfmetagenomics` for process with name output
Oct-10 18:13:37.760 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Oct-10 18:13:37.760 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Oct-10 18:13:37.762 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: handleSingleFile, pipeline:bracken, pipeline:getParams, combineFilterFastq, unpackDatabase, pipeline:kraken2, pipeline:makeReport, pipeline:getVersions, extractKraken2Reads, output, bracken, pipeline:unpackDatabase, minimap2, pipeline:unpackTaxonomy, kraken2, makeReport, checkSampleSheet, getVersions, unpackTaxonomy, extractMinimap2Reads, getParams, pipeline:combineFilterFastq
Oct-10 18:13:37.765 [main] DEBUG nextflow.Session - Ignite dataflow network (12)
Oct-10 18:13:37.770 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > handleSingleFile
Oct-10 18:13:37.771 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:unpackTaxonomy
Oct-10 18:13:37.771 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:unpackDatabase
Oct-10 18:13:37.771 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:combineFilterFastq
Oct-10 18:13:37.771 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:kraken2
Oct-10 18:13:37.773 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:bracken
Oct-10 18:13:37.773 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:getVersions
Oct-10 18:13:37.773 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:getParams
Oct-10 18:13:37.773 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > pipeline:makeReport
Oct-10 18:13:37.773 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > output
Oct-10 18:13:37.773 [main] DEBUG nextflow.script.ScriptRunner - > Await termination
Oct-10 18:13:37.773 [main] DEBUG nextflow.Session - Session await
Oct-10 18:13:37.903 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:13:37.907 [Task submitter] INFO nextflow.Session - [ae/9d5c83] Submitted process > pipeline:getVersions
Oct-10 18:13:37.927 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:13:37.927 [Task submitter] INFO nextflow.Session - [0d/eabc90] Submitted process > handleSingleFile (1)
Oct-10 18:13:37.933 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:13:37.933 [Task submitter] INFO nextflow.Session - [a9/a553a1] Submitted process > pipeline:getParams
Oct-10 18:13:38.822 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: pipeline:getParams; status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/a9/a553a1579909d694fcfe9a1b205f04]
Oct-10 18:13:38.921 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 5; name: handleSingleFile (1); status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/0d/eabc9072d79a51896a70df97dda3f3]
Oct-10 18:13:38.956 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:13:38.956 [Task submitter] INFO nextflow.Session - [9a/f7f744] Submitted process > pipeline:combineFilterFastq (1)
Oct-10 18:13:38.993 [Actor Thread 10] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=4; maxSize=4; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Oct-10 18:13:38.994 [FileTransfer-1] DEBUG nextflow.file.FilePorter - Copying foreign file https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz to work dir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/stage/41/d92233e9df42a680e09b8bc0e5a6a1/taxdump.tar.gz
Oct-10 18:13:39.533 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 4; name: pipeline:getVersions; status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/ae/9d5c8390fb6fe4569e09b0b1df3ed9]
Oct-10 18:13:40.622 [FileTransfer-2] DEBUG nextflow.file.FilePorter - Copying foreign file https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz to work dir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/stage/e7/e976cc3369c930e1a2465809ad2a36/ncbi_targeted_loci_kraken2.tar.gz
Oct-10 18:13:40.994 [Actor Thread 10] INFO nextflow.file.FilePorter - Staging foreign file: https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz
Oct-10 18:13:41.322 [FileTransfer-3] DEBUG nextflow.file.FilePorter - Copying foreign file https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib to work dir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/stage/8f/e67ffda67878d3ac4d1d15bf38f9a8/database1000mers.kmer_distrib
Oct-10 18:13:42.963 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:13:42.963 [Task submitter] INFO nextflow.Session - [06/01647c] Submitted process > pipeline:unpackTaxonomy
Oct-10 18:13:43.322 [Actor Thread 8] INFO nextflow.file.FilePorter - Staging foreign file: https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz
Oct-10 18:13:44.100 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:13:44.100 [Task submitter] INFO nextflow.Session - [25/e22fe6] Submitted process > pipeline:unpackDatabase
Oct-10 18:13:44.863 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 3; name: pipeline:unpackDatabase; status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/25/e22fe6bfc481f91441a89c8d8a902d]
Oct-10 18:13:45.358 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: pipeline:unpackTaxonomy; status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/06/01647ccd652ce35e3fc1ac27f61960]
Oct-10 18:17:40.248 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 6; name: pipeline:combineFilterFastq (1); status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/9a/f7f744e6566e82fd7b1535bce73861]
Oct-10 18:17:40.266 [Actor Thread 3] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/matteo/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/aggregate_lineages.py
Oct-10 18:17:40.274 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:17:40.274 [Task submitter] INFO nextflow.Session - [e9/821102] Submitted process > pipeline:kraken2 (1)
Oct-10 18:18:02.975 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 7; name: pipeline:kraken2 (1); status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/e9/821102f5859e3b5da6436787000118]
Oct-10 18:18:03.009 [Actor Thread 6] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/matteo/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/report-visualisation.html
Oct-10 18:18:03.010 [Actor Thread 6] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/matteo/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/report.py
Oct-10 18:18:03.015 [Actor Thread 8] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/matteo/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/run_bracken.py
Oct-10 18:18:03.017 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:18:03.018 [Task submitter] INFO nextflow.Session - [f7/24a8c1] Submitted process > pipeline:makeReport
Oct-10 18:18:03.022 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:18:03.023 [Task submitter] INFO nextflow.Session - [62/092da1] Submitted process > pipeline:bracken (1)
Oct-10 18:18:04.211 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 8; name: pipeline:bracken (1); status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/62/092da146a67a50140bb605aedfeade]
Oct-10 18:18:05.245 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 9; name: pipeline:makeReport; status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/f7/24a8c1b52af818974ed3c1f7adec5d]
Oct-10 18:18:05.263 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:18:05.263 [Task submitter] INFO nextflow.Session - [cd/d4c5dd] Submitted process > output (1)
Oct-10 18:18:05.268 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:18:05.268 [Task submitter] INFO nextflow.Session - [f9/f0fb05] Submitted process > output (7)
Oct-10 18:18:05.272 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:18:05.272 [Task submitter] INFO nextflow.Session - [ca/87f057] Submitted process > output (2)
Oct-10 18:18:05.276 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:18:05.276 [Task submitter] INFO nextflow.Session - [ef/56d261] Submitted process > output (3)
Oct-10 18:18:06.181 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 16; name: output (7); status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/f9/f0fb05b4b919887ba10dff60eff812]
Oct-10 18:18:06.184 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:18:06.184 [Task submitter] INFO nextflow.Session - [73/97e6d7] Submitted process > output (4)
Oct-10 18:18:06.212 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 11; name: output (2); status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/ca/87f057e45bb6d2dee86e893523fe37]
Oct-10 18:18:06.215 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:18:06.215 [Task submitter] INFO nextflow.Session - [e2/ced446] Submitted process > output (5)
Oct-10 18:18:06.423 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 10; name: output (1); status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/cd/d4c5dda8d9f6b4db49369716003155]
Oct-10 18:18:06.427 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:18:06.427 [Task submitter] INFO nextflow.Session - [85/9f72a5] Submitted process > output (6)
Oct-10 18:18:06.525 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 12; name: output (3); status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/ef/56d2619d3073fc6edbfc411454f64c]
Oct-10 18:18:06.527 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:18:06.527 [Task submitter] INFO nextflow.Session - [29/67faa1] Submitted process > output (8)
Oct-10 18:18:07.092 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 14; name: output (5); status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/e2/ced446a0cfbd58ff79217c0ae491d8]
Oct-10 18:18:07.094 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Oct-10 18:18:07.094 [Task submitter] INFO nextflow.Session - [23/d8520f] Submitted process > output (9)
Oct-10 18:18:07.433 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 15; name: output (6); status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/85/9f72a5ec8065228f705198e26f02eb]
Oct-10 18:18:07.452 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 13; name: output (4); status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/73/97e6d79e5bb176352655229f43caed]
Oct-10 18:18:07.502 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 17; name: output (8); status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/29/67faa1aa519beeb7f5933f76bdfd7e]
Oct-10 18:18:07.885 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 18; name: output (9); status: COMPLETED; exit: 0; error: -; workDir: /home/matteo/epi2melabs-data/nextflow/instances/2022-10-10-18-13_wf-metagenomics_5vctwqYxtrSNuvTeqT4pK3/work/23/d8520f4cfae8aa8ff7b04ca7347bbc]
Oct-10 18:18:07.887 [main] DEBUG nextflow.Session - Session await > all process finished
Oct-10 18:18:07.889 [main] DEBUG nextflow.Session - Session await > all barriers passed
Oct-10 18:18:07.898 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=18; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=4m 27s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=4; peakCpus=4; peakMemory=0; ]
Oct-10 18:18:07.898 [main] DEBUG nextflow.trace.TraceFileObserver - Flow completing -- flushing trace file
Oct-10 18:18:07.900 [main] DEBUG nextflow.trace.ReportObserver - Flow completing -- rendering html report
Oct-10 18:18:07.932 [main] DEBUG nextflow.trace.ReportObserver - Execution report summary data:
[{"cpuUsage":{"mean":88.9,"min":88.9,"q1":88.9,"q2":88.9,"q3":88.9,"max":88.9,"minLabel":"pipeline:getParams","maxLabel":"pipeline:getParams","q1Label":"pipeline:getParams","q2Label":"pipeline:getParams","q3Label":"pipeline:getParams"},"process":"getParams","mem":null,"memUsage":null,"timeUsage":null,"vmem":null,"reads":{"mean":63043,"min":63043,"q1":63043,"q2":63043,"q3":63043,"max":63043,"minLabel":"pipeline:getParams","maxLabel":"pipeline:getParams","q1Label":"pipeline:getParams","q2Label":"pipeline:getParams","q3Label":"pipeline:getParams"},"cpu":{"mean":88.9,"min":88.9,"q1":88.9,"q2":88.9,"q3":88.9,"max":88.9,"minLabel":"pipeline:getParams","maxLabel":"pipeline:getParams","q1Label":"pipeline:getParams","q2Label":"pipeline:getParams","q3Label":"pipeline:getParams"},"time":{"mean":2,"min":2,"q1":2,"q2":2,"q3":2,"max":2,"minLabel":"pipeline:getParams","maxLabel":"pipeline:getParams","q1Label":"pipeline:getParams","q2Label":"pipeline:getParams","q3Label":"pipeline:getParams"},"writes":{"mean":3390,"min":3390,"q1":3390,"q2":3390,"q3":3390,"max":3390,"minLabel":"pipeline:getParams","maxLabel":"pipeline:getParams","q1Label":"pipeline:getParams","q2Label":"pipeline:getParams","q3Label":"pipeline:getParams"}},{"cpuUsage":null,"process":"handleSingleFile","mem":null,"memUsage":null,"timeUsage":null,"vmem":null,"reads":{"mean":77818,"min":77818,"q1":77818,"q2":77818,"q3":77818,"max":77818,"minLabel":"handleSingleFile (1)","maxLabel":"handleSingleFile (1)","q1Label":"handleSingleFile (1)","q2Label":"handleSingleFile (1)","q3Label":"handleSingleFile (1)"},"cpu":null,"time":{"mean":4,"min":4,"q1":4,"q2":4,"q3":4,"max":4,"minLabel":"handleSingleFile (1)","maxLabel":"handleSingleFile (1)","q1Label":"handleSingleFile (1)","q2Label":"handleSingleFile (1)","q3Label":"handleSingleFile (1)"},"writes":{"mean":179,"min":179,"q1":179,"q2":179,"q3":179,"max":179,"minLabel":"handleSingleFile (1)","maxLabel":"handleSingleFile (1)","q1Label":"handleSingleFile (1)","q2Label":"handleSingleFile (1)","q3Label":"handleSingleFile (1)"}},{"cpuUsage":{"mean":81.2,"min":81.2,"q1":81.2,"q2":81.2,"q3":81.2,"max":81.2,"minLabel":"pipeline:getVersions","maxLabel":"pipeline:getVersions","q1Label":"pipeline:getVersions","q2Label":"pipeline:getVersions","q3Label":"pipeline:getVersions"},"process":"getVersions","mem":{"mean":9203712,"min":9203712,"q1":9203712,"q2":9203712,"q3":9203712,"max":9203712,"minLabel":"pipeline:getVersions","maxLabel":"pipeline:getVersions","q1Label":"pipeline:getVersions","q2Label":"pipeline:getVersions","q3Label":"pipeline:getVersions"},"memUsage":null,"timeUsage":null,"vmem":{"mean":13451264,"min":13451264,"q1":13451264,"q2":13451264,"q3":13451264,"max":13451264,"minLabel":"pipeline:getVersions","maxLabel":"pipeline:getVersions","q1Label":"pipeline:getVersions","q2Label":"pipeline:getVersions","q3Label":"pipeline:getVersions"},"reads":{"mean":10234543,"min":10234543,"q1":10234543,"q2":10234543,"q3":10234543,"max":10234543,"minLabel":"pipeline:getVersions","maxLabel":"pipeline:getVersions","q1Label":"pipeline:getVersions","q2Label":"pipeline:getVersions","q3Label":"pipeline:getVersions"},"cpu":{"mean":81.2,"min":81.2,"q1":81.2,"q2":81.2,"q3":81.2,"max":81.2,"minLabel":"pipeline:getVersions","maxLabel":"pipeline:getVersions","q1Label":"pipeline:getVersions","q2Label":"pipeline:getVersions","q3Label":"pipeline:getVersions"},"time":{"mean":669,"min":669,"q1":669,"q2":669,"q3":669,"max":669,"minLabel":"pipeline:getVersions","maxLabel":"pipeline:getVersions","q1Label":"pipeline:getVersions","q2Label":"pipeline:getVersions","q3Label":"pipeline:getVersions"},"writes":{"mean":7508,"min":7508,"q1":7508,"q2":7508,"q3":7508,"max":7508,"minLabel":"pipeline:getVersions","maxLabel":"pipeline:getVersions","q1Label":"pipeline:getVersions","q2Label":"pipeline:getVersions","q3Label":"pipeline:getVersions"}},{"cpuUsage":{"mean":83.7,"min":83.7,"q1":83.7,"q2":83.7,"q3":83.7,"max":83.7,"minLabel":"pipeline:unpackDatabase","maxLabel":"pipeline:unpackDatabase","q1Label":"pipeline:unpackDatabase","q2Label":"pipeline:unpackDatabase","q3Label":"pipeline:unpackDatabase"},"process":"unpackDatabase","mem":{"mean":6709248,"min":6709248,"q1":6709248,"q2":6709248,"q3":6709248,"max":6709248,"minLabel":"pipeline:unpackDatabase","maxLabel":"pipeline:unpackDatabase","q1Label":"pipeline:unpackDatabase","q2Label":"pipeline:unpackDatabase","q3Label":"pipeline:unpackDatabase"},"memUsage":null,"timeUsage":null,"vmem":{"mean":11759616,"min":11759616,"q1":11759616,"q2":11759616,"q3":11759616,"max":11759616,"minLabel":"pipeline:unpackDatabase","maxLabel":"pipeline:unpackDatabase","q1Label":"pipeline:unpackDatabase","q2Label":"pipeline:unpackDatabase","q3Label":"pipeline:unpackDatabase"},"reads":{"mean":22909412,"min":22909412,"q1":22909412,"q2":22909412,"q3":22909412,"max":22909412,"minLabel":"pipeline:unpackDatabase","maxLabel":"pipeline:unpackDatabase","q1Label":"pipeline:unpackDatabase","q2Label":"pipeline:unpackDatabase","q3Label":"pipeline:unpackDatabase"},"cpu":{"mean":83.7,"min":83.7,"q1":83.7,"q2":83.7,"q3":83.7,"max":83.7,"minLabel":"pipeline:unpackDatabase","maxLabel":"pipeline:unpackDatabase","q1Label":"pipeline:unpackDatabase","q2Label":"pipeline:unpackDatabase","q3Label":"pipeline:unpackDatabase"},"time":{"mean":108,"min":108,"q1":108,"q2":108,"q3":108,"max":108,"minLabel":"pipeline:unpackDatabase","maxLabel":"pipeline:unpackDatabase","q1Label":"pipeline:unpackDatabase","q2Label":"pipeline:unpackDatabase","q3Label":"pipeline:unpackDatabase"},"writes":{"mean":27683015,"min":27683015,"q1":27683015,"q2":27683015,"q3":27683015,"max":27683015,"minLabel":"pipeline:unpackDatabase","maxLabel":"pipeline:unpackDatabase","q1Label":"pipeline:unpackDatabase","q2Label":"pipeline:unpackDatabase","q3Label":"pipeline:unpackDatabase"}},{"cpuUsage":{"mean":97.2,"min":97.2,"q1":97.2,"q2":97.2,"q3":97.2,"max":97.2,"minLabel":"pipeline:unpackTaxonomy","maxLabel":"pipeline:unpackTaxonomy","q1Label":"pipeline:unpackTaxonomy","q2Label":"pipeline:unpackTaxonomy","q3Label":"pipeline:unpackTaxonomy"},"process":"unpackTaxonomy","mem":{"mean":6713344,"min":6713344,"q1":6713344,"q2":6713344,"q3":6713344,"max":6713344,"minLabel":"pipeline:unpackTaxonomy","maxLabel":"pipeline:unpackTaxonomy","q1Label":"pipeline:unpackTaxonomy","q2Label":"pipeline:unpackTaxonomy","q3Label":"pipeline:unpackTaxonomy"},"memUsage":null,"timeUsage":null,"vmem":{"mean":11759616,"min":11759616,"q1":11759616,"q2":11759616,"q3":11759616,"max":11759616,"minLabel":"pipeline:unpackTaxonomy","maxLabel":"pipeline:unpackTaxonomy","q1Label":"pipeline:unpackTaxonomy","q2Label":"pipeline:unpackTaxonomy","q3Label":"pipeline:unpackTaxonomy"},"reads":{"mean":475380345,"min":475380345,"q1":475380345,"q2":475380345,"q3":475380345,"max":475380345,"minLabel":"pipeline:unpackTaxonomy","maxLabel":"pipeline:unpackTaxonomy","q1Label":"pipeline:unpackTaxonomy","q2Label":"pipeline:unpackTaxonomy","q3Label":"pipeline:unpackTaxonomy"},"cpu":{"mean":97.2,"min":97.2,"q1":97.2,"q2":97.2,"q3":97.2,"max":97.2,"minLabel":"pipeline:unpackTaxonomy","maxLabel":"pipeline:unpackTaxonomy","q1Label":"pipeline:unpackTaxonomy","q2Label":"pipeline:unpackTaxonomy","q3Label":"pipeline:unpackTaxonomy"},"time":{"mean":1837,"min":1837,"q1":1837,"q2":1837,"q3":1837,"max":1837,"minLabel":"pipeline:unpackTaxonomy","maxLabel":"pipeline:unpackTaxonomy","q1Label":"pipeline:unpackTaxonomy","q2Label":"pipeline:unpackTaxonomy","q3Label":"pipeline:unpackTaxonomy"},"writes":{"mean":832433111,"min":832433111,"q1":832433111,"q2":832433111,"q3":832433111,"max":832433111,"minLabel":"pipeline:unpackTaxonomy","maxLabel":"pipeline:unpackTaxonomy","q1Label":"pipeline:unpackTaxonomy","q2Label":"pipeline:unpackTaxonomy","q3Label":"pipeline:unpackTaxonomy"}},{"cpuUsage":{"mean":97.9,"min":97.9,"q1":97.9,"q2":97.9,"q3":97.9,"max":97.9,"minLabel":"pipeline:combineFilterFastq (1)","maxLabel":"pipeline:combineFilterFastq (1)","q1Label":"pipeline:combineFilterFastq (1)","q2Label":"pipeline:combineFilterFastq (1)","q3Label":"pipeline:combineFilterFastq (1)"},"process":"combineFilterFastq","mem":{"mean":8605696,"min":8605696,"q1":8605696,"q2":8605696,"q3":8605696,"max":8605696,"minLabel":"pipeline:combineFilterFastq (1)","maxLabel":"pipeline:combineFilterFastq (1)","q1Label":"pipeline:combineFilterFastq (1)","q2Label":"pipeline:combineFilterFastq (1)","q3Label":"pipeline:combineFilterFastq (1)"},"memUsage":null,"timeUsage":null,"vmem":{"mean":12800000,"min":12800000,"q1":12800000,"q2":12800000,"q3":12800000,"max":12800000,"minLabel":"pipeline:combineFilterFastq (1)","maxLabel":"pipeline:combineFilterFastq (1)","q1Label":"pipeline:combineFilterFastq (1)","q2Label":"pipeline:combineFilterFastq (1)","q3Label":"pipeline:combineFilterFastq (1)"},"reads":{"mean":16257589988,"min":16257589988,"q1":16257589988,"q2":16257589988,"q3":16257589988,"max":16257589988,"minLabel":"pipeline:combineFilterFastq (1)","maxLabel":"pipeline:combineFilterFastq (1)","q1Label":"pipeline:combineFilterFastq (1)","q2Label":"pipeline:combineFilterFastq (1)","q3Label":"pipeline:combineFilterFastq (1)"},"cpu":{"mean":97.9,"min":97.9,"q1":97.9,"q2":97.9,"q3":97.9,"max":97.9,"minLabel":"pipeline:combineFilterFastq (1)","maxLabel":"pipeline:combineFilterFastq (1)","q1Label":"pipeline:combineFilterFastq (1)","q2Label":"pipeline:combineFilterFastq (1)","q3Label":"pipeline:combineFilterFastq (1)"},"time":{"mean":240684,"min":240684,"q1":240684,"q2":240684,"q3":240684,"max":240684,"minLabel":"pipeline:combineFilterFastq (1)","maxLabel":"pipeline:combineFilterFastq (1)","q1Label":"pipeline:combineFilterFastq (1)","q2Label":"pipeline:combineFilterFastq (1)","q3Label":"pipeline:combineFilterFastq (1)"},"writes":{"mean":103433582,"min":103433582,"q1":103433582,"q2":103433582,"q3":103433582,"max":103433582,"minLabel":"pipeline:combineFilterFastq (1)","maxLabel":"pipeline:combineFilterFastq (1)","q1Label":"pipeline:combineFilterFastq (1)","q2Label":"pipeline:combineFilterFastq (1)","q3Label":"pipeline:combineFilterFastq (1)"}},{"cpuUsage":{"mean":92.3,"min":92.3,"q1":92.3,"q2":92.3,"q3":92.3,"max":92.3,"minLabel":"pipeline:kraken2 (1)","maxLabel":"pipeline:kraken2 (1)","q1Label":"pipeline:kraken2 (1)","q2Label":"pipeline:kraken2 (1)","q3Label":"pipeline:kraken2 (1)"},"process":"kraken2","mem":{"mean":739254272,"min":739254272,"q1":739254272,"q2":739254272,"q3":739254272,"max":739254272,"minLabel":"pipeline:kraken2 (1)","maxLabel":"pipeline:kraken2 (1)","q1Label":"pipeline:kraken2 (1)","q2Label":"pipeline:kraken2 (1)","q3Label":"pipeline:kraken2 (1)"},"memUsage":null,"timeUsage":null,"vmem":{"mean":1514602496,"min":1514602496,"q1":1514602496,"q2":1514602496,"q3":1514602496,"max":1514602496,"minLabel":"pipeline:kraken2 (1)","maxLabel":"pipeline:kraken2 (1)","q1Label":"pipeline:kraken2 (1)","q2Label":"pipeline:kraken2 (1)","q3Label":"pipeline:kraken2 (1)"},"reads":{"mean":511901571,"min":511901571,"q1":511901571,"q2":511901571,"q3":511901571,"max":511901571,"minLabel":"pipeline:kraken2 (1)","maxLabel":"pipeline:kraken2 (1)","q1Label":"pipeline:kraken2 (1)","q2Label":"pipeline:kraken2 (1)","q3Label":"pipeline:kraken2 (1)"},"cpu":{"mean":92.3,"min":92.3,"q1":92.3,"q2":92.3,"q3":92.3,"max":92.3,"minLabel":"pipeline:kraken2 (1)","maxLabel":"pipeline:kraken2 (1)","q1Label":"pipeline:kraken2 (1)","q2Label":"pipeline:kraken2 (1)","q3Label":"pipeline:kraken2 (1)"},"time":{"mean":21964,"min":21964,"q1":21964,"q2":21964,"q3":21964,"max":21964,"minLabel":"pipeline:kraken2 (1)","maxLabel":"pipeline:kraken2 (1)","q1Label":"pipeline:kraken2 (1)","q2Label":"pipeline:kraken2 (1)","q3Label":"pipeline:kraken2 (1)"},"writes":{"mean":102627382,"min":102627382,"q1":102627382,"q2":102627382,"q3":102627382,"max":102627382,"minLabel":"pipeline:kraken2 (1)","maxLabel":"pipeline:kraken2 (1)","q1Label":"pipeline:kraken2 (1)","q2Label":"pipeline:kraken2 (1)","q3Label":"pipeline:kraken2 (1)"}},{"cpuUsage":{"mean":94,"min":94,"q1":94,"q2":94,"q3":94,"max":94,"minLabel":"pipeline:bracken (1)","maxLabel":"pipeline:bracken (1)","q1Label":"pipeline:bracken (1)","q2Label":"pipeline:bracken (1)","q3Label":"pipeline:bracken (1)"},"process":"bracken","mem":{"mean":11300864,"min":11300864,"q1":11300864,"q2":11300864,"q3":11300864,"max":11300864,"minLabel":"pipeline:bracken (1)","maxLabel":"pipeline:bracken (1)","q1Label":"pipeline:bracken (1)","q2Label":"pipeline:bracken (1)","q3Label":"pipeline:bracken (1)"},"memUsage":null,"timeUsage":null,"vmem":{"mean":15327232,"min":15327232,"q1":15327232,"q2":15327232,"q3":15327232,"max":15327232,"minLabel":"pipeline:bracken (1)","maxLabel":"pipeline:bracken (1)","q1Label":"pipeline:bracken (1)","q2Label":"pipeline:bracken (1)","q3Label":"pipeline:bracken (1)"},"reads":{"mean":10608078,"min":10608078,"q1":10608078,"q2":10608078,"q3":10608078,"max":10608078,"minLabel":"pipeline:bracken (1)","maxLabel":"pipeline:bracken (1)","q1Label":"pipeline:bracken (1)","q2Label":"pipeline:bracken (1)","q3Label":"pipeline:bracken (1)"},"cpu":{"mean":94,"min":94,"q1":94,"q2":94,"q3":94,"max":94,"minLabel":"pipeline:bracken (1)","maxLabel":"pipeline:bracken (1)","q1Label":"pipeline:bracken (1)","q2Label":"pipeline:bracken (1)","q3Label":"pipeline:bracken (1)"},"time":{"mean":520,"min":520,"q1":520,"q2":520,"q3":520,"max":520,"minLabel":"pipeline:bracken (1)","maxLabel":"pipeline:bracken (1)","q1Label":"pipeline:bracken (1)","q2Label":"pipeline:bracken (1)","q3Label":"pipeline:bracken (1)"},"writes":{"mean":5651,"min":5651,"q1":5651,"q2":5651,"q3":5651,"max":5651,"minLabel":"pipeline:bracken (1)","maxLabel":"pipeline:bracken (1)","q1Label":"pipeline:bracken (1)","q2Label":"pipeline:bracken (1)","q3Label":"pipeline:bracken (1)"}},{"cpuUsage":{"mean":90.1,"min":90.1,"q1":90.1,"q2":90.1,"q3":90.1,"max":90.1,"minLabel":"pipeline:makeReport","maxLabel":"pipeline:makeReport","q1Label":"pipeline:makeReport","q2Label":"pipeline:makeReport","q3Label":"pipeline:makeReport"},"process":"makeReport","mem":{"mean":123756544,"min":123756544,"q1":123756544,"q2":123756544,"q3":123756544,"max":123756544,"minLabel":"pipeline:makeReport","maxLabel":"pipeline:makeReport","q1Label":"pipeline:makeReport","q2Label":"pipeline:makeReport","q3Label":"pipeline:makeReport"},"memUsage":null,"timeUsage":null,"vmem":{"mean":1172783104,"min":1172783104,"q1":1172783104,"q2":1172783104,"q3":1172783104,"max":1172783104,"minLabel":"pipeline:makeReport","maxLabel":"pipeline:makeReport","q1Label":"pipeline:makeReport","q2Label":"pipeline:makeReport","q3Label":"pipeline:makeReport"},"reads":{"mean":52415495,"min":52415495,"q1":52415495,"q2":52415495,"q3":52415495,"max":52415495,"minLabel":"pipeline:makeReport","maxLabel":"pipeline:makeReport","q1Label":"pipeline:makeReport","q2Label":"pipeline:makeReport","q3Label":"pipeline:makeReport"},"cpu":{"mean":90.1,"min":90.1,"q1":90.1,"q2":90.1,"q3":90.1,"max":90.1,"minLabel":"pipeline:makeReport","maxLabel":"pipeline:makeReport","q1Label":"pipeline:makeReport","q2Label":"pipeline:makeReport","q3Label":"pipeline:makeReport"},"time":{"mean":1434,"min":1434,"q1":1434,"q2":1434,"q3":1434,"max":1434,"minLabel":"pipeline:makeReport","maxLabel":"pipeline:makeReport","q1Label":"pipeline:makeReport","q2Label":"pipeline:makeReport","q3Label":"pipeline:makeReport"},"writes":{"mean":3617050,"min":3617050,"q1":3617050,"q2":3617050,"q3":3617050,"max":3617050,"minLabel":"pipeline:makeReport","maxLabel":"pipeline:makeReport","q1Label":"pipeline:makeReport","q2Label":"pipeline:makeReport","q3Label":"pipeline:makeReport"}},{"cpuUsage":{"mean":152.37,"min":114.3,"q1":133.3,"q2":133.3,"q3":183.33,"max":200,"minLabel":"output (9)","maxLabel":"output (4)","q1Label":"output (7)","q2Label":"output (5)","q3Label":"output (8)"},"process":"output","mem":null,"memUsage":null,"timeUsage":null,"vmem":null,"reads":{"mean":59765,"min":59762,"q1":59763,"q2":59766,"q3":59766,"max":59767,"minLabel":"output (2)","maxLabel":"output (4)","q1Label":"output (3)","q2Label":"output (6)","q3Label":"output (9)"},"cpu":{"mean":101.58,"min":0,"q1":0,"q2":133.3,"q3":133.3,"max":200,"minLabel":"output (2)","maxLabel":"output (4)","q1Label":"output (3)","q2Label":"output (7)","q3Label":"output (8)"},"time":{"mean":2.11,"min":2,"q1":2,"q2":2,"q3":2,"max":3,"minLabel":"output (7)","maxLabel":"output (4)","q1Label":"output (1)","q2Label":"output (5)","q3Label":"output (8)"},"writes":{"mean":202.67,"min":200,"q1":200,"q2":204,"q3":204,"max":204,"minLabel":"output (2)","maxLabel":"output (9)","q1Label":"output (3)","q2Label":"output (5)","q3Label":"output (4)"}}]
Oct-10 18:18:08.358 [main] DEBUG nextflow.trace.TimelineObserver - Flow completing -- rendering html timeline
Oct-10 18:18:08.416 [main] WARN nextflow.dag.GraphvizRenderer - To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.
Oct-10 18:18:08.420 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Oct-10 18:18:08.426 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye[Bug]: Automatic finding of barcoded files not working when using minimap2 classifier
What happened?
This bug is linked to specifying a directory which holds directories which hold fastq(.gz) files, as an argument. It is not an issue when specifying a full path to a fastq(.gz) file.
The folder structure is
> ls -1 guppy_barcoder/
barcode01
barcode02
barcode03
barcode04
barcode06
barcode07
barcode08
barcode09
barcode10
barcode11
barcode12
barcode13
barcode14
barcode15
barcode16
barcode17
barcode18
barcode19
barcode20
barcode21
barcoding_summary.txt
guppy_barcoder-core-dump-db
guppy_barcoder_log-2023-02-16_09-45-02.log
unclassified
> ls -1 guppy_barcoder/barcode01 | head
fastq_runid_72506084f08d36ab1768237c543c2d96290fe7d9_0.fastq.gz
fastq_runid_72506084f08d36ab1768237c543c2d96290fe7d9_10.fastq.gz
fastq_runid_72506084f08d36ab1768237c543c2d96290fe7d9_11.fastq.gz
fastq_runid_72506084f08d36ab1768237c543c2d96290fe7d9_12.fastq.gz
fastq_runid_72506084f08d36ab1768237c543c2d96290fe7d9_13.fastq.gz
fastq_runid_72506084f08d36ab1768237c543c2d96290fe7d9_14.fastq.gz
fastq_runid_72506084f08d36ab1768237c543c2d96290fe7d9_15.fastq.gz
fastq_runid_72506084f08d36ab1768237c543c2d96290fe7d9_16.fastq.gz
fastq_runid_72506084f08d36ab1768237c543c2d96290fe7d9_17.fastq.gz
fastq_runid_72506084f08d36ab1768237c543c2d96290fe7d9_18.fastq.gz
When I run
./nextflow run epi2me-labs/wf-metagenomics -profile singularity --out_dir all_bcs_minimap2 --fastq $PREFIX/guppy_barcoder/ --classifier minimap2 # ${PREFIX} is some path
on our HPC (https://hpc.uni.lu; Rocky Linux 8.7 (Green Obsidian)), the workflow is unable to identify the fastq.gz files, i.e., it exits with None of the directories given contain .fastq(.gz) files.
In contrast, when I use the kraken2 classifier, with
./nextflow run epi2me-labs/wf-metagenomics -profile singularity --out_dir all_bcs_minimap2 --fastq $PREFIX/guppy_barcoder/ --classifier kraken2 # ${PREFIX} is some path
It works fine and the HTML report nicely aggregates the read counts by barcode, just as in https://labs.epi2me.io/workflows/wf-metagenomics-report.html.
I thus had a bit of a look but as this is the very first time I am dealing with nextflow, I might have missed something.
In the fastqingress code, specifically
wf-metagenomics/lib/fastqingress.nf
Line 177 in af0af1b
find_fastq call is false. However, that prohibits it from descending one further folder, i.e., into the individual barcode folders to find the fastq.gz files there (output from guppy_barcoder).So
search_subdirs is false at wf-metagenomics/lib/fastqingress.nf
Line 100 in af0af1b
files is thus not filled with fastq.gz files.
Due to the real-time watch functionality that is exclusive to the kraken2 classifier, it seems that this is differently solved for kraken2, but I did not understand how.
Either way, is this indeed a bug?
If so, how could this be fixed?
Thanks a lot!
Best wishes and stay safe,
Cedric
P.S. I actually have to specify also a --port, e.g., 9123, when running the kraken2 classifier as the default port 8080 is not available. This was killing the kraken2_server but I could not find a message in the .nextflow.log. Only by installing it via mamba and trying to run it manually, I received error messages that the port was already in use.
This might be relevant for other where it seems that the kraken2 classification is stalled.
I omitted the --port 9123 argument here for the sake of simplicity.
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
Singularity
Workflow Version
v2.0.8-g3e8fdcd.
Relevant log output
Output Options
out_dir : all_bcs_minimap2
Advanced Options
threads : 12
Other parameters
process_label : wfmetagenomics
!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-metagenomics for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
--------------------------------------------------------------------------------
This is epi2me-labs/wf-metagenomics v2.0.8-g3e8fdcd.
--------------------------------------------------------------------------------
Checking inputs.
Checking fastq input.
Barcoded directories detected.
Non barcoded directories detected.
None of the directories given contain .fastq(.gz) files.[Bug]: kraken_pipeline.nf - taxonkit could not find lineage
What happened?
Lines 139 and 140 of kraken_pipeline.nf
you are assuming that every OTUs have 2 names, but what about this OTU name?
Severe acute respiratory syndrome coronavirus
field $3 will be "respiratory" and not the taxonomy_id... so I sugest this (use tab as separator in awk command instead of space):
awk -F'\t' '{ print \$2,\$6}' "${sample_id}.bracken_report.txt" | awk 'NR!=1 {print}' > taxacounts.txt
awk -F'\t' '{print \$2}' "${sample_id}.bracken_report.txt" | awk 'NR!=1 {print}' > taxa.txt
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
ontresearch/wf-metagenomics
Workflow Execution - Execution Profile
Docker
Workflow Version
v2.0.1
Relevant log output
Already solved the bug in my local workflow (cloned form your repo) and lost the logs but the error was in bracken nextflow process in the taxonkit command: it could not find the lineage "respiratory"[Bug]: No output file with classified and unclassified reads
What happened?
In the previous pipeline version (for example v1.1.0) one of the outputs was a classified reads fastq file with details about classification of each read. Is it possible to retain this feature as it is very useful for downstream analysis or at least make it optional if by default you are trying to avoid large data accumulation?
This refers to --classified-out option in kaken2
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
Singularity
Workflow Version
2.0.8
Relevant log output
not relevantSample sheet not recognized by workflow
I have been able to run the workflow with a single sample and have the workflow rename the output file with the sample ID, and I have also been able to run it with barcoded multiple samples with the output files labeled with barcode # (i.e. barcode01.braken_report.txt) with no issues. However, when I try to specify a sample sheet with sample ID and barcode #, the workflow fails to run and I get the below error:
_Error executing process > 'checkSampleSheet'
Caused by:
Process checkSampleSheet terminated with an error exit status (1)
Command executed:
check_sample_sheet.py sample_sheet.txt samples.txt
Command exit status:
1
Command output:
(empty)
Command error:
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/util/decorators.py:311: ParserWarning: Falling back to the 'python' engine because the 'c' eng
ine does not support sep=None with delim_whitespace=False; you can avoid this warning by specifying engine='python'.
return func(*args, **kwargs)
Traceback (most recent call last):
File "/Users/JohnSmith/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/check_sample_sheet.py", line 26, in main
raise IOError()
OSError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/JohnSmith/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/check_sample_sheet.py", line 41, in
main()
File "/Users/JohnSmith/.nextflow/assets/epi2me-labs/wf-metagenomics/bin/check_sample_sheet.py", line 28, in main
raise IOError(
OSError: Could not parse sample sheet, it must contain two columns named 'barcode' and 'sample_id' or 'alias'.
Work dir:
/Users/JohnSmith/Desktop/EPI2ME_wf-metagenomics/work/c3/f25acbd408f0e16f71c1be7dd2ca1f
Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line
It seems the description of the sample sheet CSV file is wrong. It says to contain three columns: CSV file with columns named barcode, sample_name and type. Permissible if passing a directory containing barcodeXX sub-directories.
but this gives an error: OSError: Could not parse sample sheet, it must contain two columns named 'barcode' and 'sample_id' or 'alias'.
How can I fix the format of this sample sheet so that the workflow recognizes the sample IDs and labels the output files properly?
Permission Denied
What happened?
I ran the workflow of wf-metagenomics on epi2melabs in ubuntu 22.04.
I encountered a permission issue: touch: cannot touch '.command.trace': Permission denied
I had already run sudo usermod -aG docker $USER after installing docker and checked it can work without sudo.
I'm new to coding, so I'd tried my best to understand this issue beforehand.
I found out that adding docker.runOptions = '-u $(id -u):$(id -g)' in the nextflow.config seemed to solve this problem.
However, I saw a similar code in the nextflow.config. But I'm not sure whether that code helped, so I added docker.runOptions = '-u $(id -u):$(id -g)' in nextflow.config anyways, but it still didn't work.
Even I remove what I added, it still didn't work.
I also tried to run the docker image directly, but it exited immediately, (I'm not sure whether this information was helpful or not.)
Additionally, since I ran barcoded samples, I tried to run each barcode individually trying to circumvent the potential problem of "too much data", but of course, it still didn't work.
What should I do to solve this permission problem? Or there were other issues that I didn't aware of?
Thanks in advance.
Operating System
ubuntu 20.04
Workflow Execution
EPI2ME Labs desktop application
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
None
Workflow Version
2.0.8
Relevant log output
Error executing process > 'kraken_pipeline:getParams'
Caused by:
Process `kraken_pipeline:getParams` terminated with an error exit status (1)
Command executed:
# Output nextflow params object to JSON
echo '{
"help": false,
"version": false,
"fastq": "/media/ac307/R111/R11623009/nanopore/16S_1_fastq_pass/barcode01",
"sample": null,
"sample_sheet": null,
"max_len": null,
"min_len": 0,
"taxonomy": null,
"classifier": "kraken2",
"reference": null,
"ref2taxid": null,
"minimap2filter": null,
"minimap2exclude": false,
"split_prefix": false,
"database": null,
"bracken_dist": null,
"bracken_length": null,
"bracken_level": "S",
"out_dir": "/home/ac307/epi2melabs/instances/wf-metagenomics_538a4fd5-1e07-4eaf-8b3f-29d6477ccf14/output",
"wfversion": "v2.0.8",
"disable_ping": false,
"threads": 2,
"aws_image_prefix": null,
"aws_queue": null,
"batch_size": 0,
"watch_path": false,
"store_dir": "store_dir",
"read_limit": null,
"port": 8080,
"database_set": "ncbi_16s_18s",
"database_sets": {
"ncbi_16s_18s": {
"reference": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna",
"refindex": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna.fai",
"database": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz",
"kmer_dist": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib",
"ref2taxid": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ref2taxid.targloci.tsv",
"taxonomy": "https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz"
},
"ncbi_16s_18s_28s_ITS": {
"reference": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna",
"refindex": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna.fai",
"database": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS_kraken2.tar.gz",
"kmer_dist": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/database1000mers.kmer_distrib",
"ref2taxid": "https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ref2taxid.ncbi_16s_18s_28s_ITS.tsv",
"taxonomy": "https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz"
},
"PlusPF-8": {
"database": "https://genome-idx.s3.amazonaws.com/kraken/k2_pluspf_8gb_20210517.tar.gz",
"taxonomy": "https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz"
}
},
"process_label": "wfmetagenomics",
"monochrome_logs": false,
"validate_params": true,
"show_hidden_params": false,
"analyse_unclassified": false,
"schema_ignore_params": "show_hidden_params,validate_params,monochrome_logs,aws_queue,aws_image_prefix,pangolin_version,wfversion,wf,process_label",
"wf": {
"example_cmd": [
"--fastq test_data/barcode01/reads.fastq.gz"
],
"agent": null,
"container_sha": "sha156f44a25dd22e39eb10e82d12e0584d41dde909"
},
"wf.agent": "epi2melabs/4.1.1"
}' > params.json
Command exit status:
1
Command output:
(empty)
Command error:
touch: cannot touch '.command.trace': Permission denied
Work dir:
/home/ac307/epi2melabs/instances/wf-metagenomics_538a4fd5-1e07-4eaf-8b3f-29d6477ccf14/work/31/0a716a95c0dd9dd801bd5e068313ba
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.Process `handleSingleFile (1)` and `pipeline:getVersions` terminated with an error exit status (127)
Hi, when i run:
nextflow run epi2me-labs/wf-metagenomics/ --fastq /mnt/lustre/scratch/nlsas/home/csic/nmy/vgc/dataANDanalysis/analysis/16s/16s_initial_5samples/16s_quality_length_filter.fastq --threads 64 --source ncbi_16s_18s
It raises the next error :
> Error executing process > 'pipeline:getVersions'
>
> Caused by:
> Process `pipeline:getVersions` terminated with an error exit status (127)
>
> Command executed:
>
> python -c "import pysam; print(f'pysam,{pysam.__version__}')" >> versions.txt
> python -c "import pandas; print(f'pandas,{pandas.__version__}')" >> versions.txt
> fastcat --version | sed 's/^/fastcat,/' >> versions.txt
> minimap2 --version | sed 's/^/minimap2,/' >> versions.txt
> samtools --version | head -n 1 | sed 's/ /,/' >> versions.txt
> taxonkit version | sed 's/ /,/' >> versions.txt
> kraken2 --version | head -n 1 | sed 's/ version /,/' >> versions.txt
>
> Command exit status:
> 127
>
> Command output:
> (empty)
>
> Command error:
> .command.run: line 279: docker: command not found
>
> Work dir:
> /mnt/netapp1/Store_CSIC/home/csic/nmy/vgc/metagenomics/nextflow/epi2me-labs/wf-metagenomics/work/7a/361c9fa3737593405
>
> Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
>
>
I tryed with:
nextflow run epi2me-labs/wf-metagenomics/ --fastq /mnt/lustre/scratch/nls16s/16s_initial_5samples/16s_quality_length_filter.fastq --minimap2 --threads 64
But a similar error appears:
> Error executing process > 'handleSingleFile (1)'
>
> Caused by:
> Process `handleSingleFile (1)` terminated with an error exit status (127)
>
> Command executed:
>
> mkdir 16s_quality_length_filter
> mv 16s_quality_length_filter.fastq 16s_quality_length_filter
>
> Command exit status:
> 127
>
> Command output:
> (empty)
>
> Command error:
> .command.run: line 279: docker: command not found
>
> Work dir:
> /mnt/netapp1/Store_CSIC/home/csic/nmy/vgc/metagenomics/nextflow/epi2me-labs/wf-metagenomics/work/e5/75cd9ea0f88868016eabdd546378f8
>
> Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
>
>
I tried with my own minimap reference and it raises the same
Im in a cluster, so i changed executor to slurm in nextflow.config as documentation in nextflow suggests, but it raises grid scheduler error. Lastly i have changed executor specifications to cpu=64 and ram to 64 GB in order to set the same specifications of the salloc which i ask for in the cluster. But the error doesnt change.
My nexflow version is 22.04.0
Do you figure out what could be happening?
Thank you in advanced
Best,
Sergio
[Bug]: wf-metagenomics keeps failing
What happened?
When I use the demo data everything works, but when I try to run my own data it keeps failing with errors and also notes that I need to install graphviz however graphviz is already installed. Log output below
Operating System
Windows 11
Workflow Execution
EPI2ME Labs desktop application
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
Docker
Workflow Version
V4.11
Relevant log output
ry
Resources
Tasks
[admiring_lovelace]
Nextflow workflow report
[admiring_lovelace]
Workflow execution completed unsuccessfully!
The exit status of the task that caused the workflow execution to fail was: 2.
The full error message was:
Error executing process > 'kraken_pipeline:progressive_bracken (1)'
Caused by:
Process `kraken_pipeline:progressive_bracken (1)` terminated with an error exit status (2)
Command executed:
# run bracken on the latest kreports, is this writing some outputs
# alongside the inputs? seems at least {}.kreport_bracken_species.txt
# is written alongside the input
run_bracken.py \
"database_dir" \
"kraken.1.unclassified/unclassified.kreport.txt" \
"1000" \
"S" \
"unclassified.bracken_report.txt"
# do some stuff...
awk -F ' ' -v OFS=' ' '{ print $2,$6 }' "unclassified.bracken_report.txt" \
| awk -F ' ' -v OFS=' ' 'NR!=1 {print}' \
| tee taxacounts.txt \
| awk -F ' ' -v OFS=' ' '{ print $1 }' > taxa.txt
taxonkit lineage -j 2 \
--data-dir taxonomy_dir \
-R taxa.txt > lineages.txt
aggregate_lineages_bracken.py \
-i "lineages.txt" -b "taxacounts.txt" \
-u "kraken.1.unclassified/unclassified.kreport.txt" \
-p "unclassified.kraken2"
file1=`cat *.json`
echo "{"'"unclassified"'": "$file1"}" >> "bracken.json"
# collate the latest bracken outputs into state
if [[ "1" != "1" ]]; then
cp -r "NOSTATE" "bracken.1"
else
# make fresh directory
mkdir "bracken.1"
fi;
# first output here is just for end user
mv "kraken.1.unclassified/unclassified.kreport_bracken_species.txt" "bracken.1" || echo "No bracken report"
mv "bracken.json" "bracken.1/unclassified.json"
Command exit status:
2
Command output:
b' >> Checking for Valid Options...\n ERROR: database_dir/database1000mers.kmer_distrib does not exist\n Run bracken-build to generate the kmer distribution file.\n'
Command error:
b''awk: cannot open unclassified.bracken_report.txt (No such file or directory)
Work dir:
/mnt/c/Users/meaga/epi2melabs/instances/wf-metagenomics_f7c179f2-5dae-473d-983d-6ea1c524321c/work/ca/688912bc1e50127ca1e628c3c3fa0b
Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`[Bug]: kraken2-client fails to connect to kraken2-server
What happened?
When I run the workflow it doesn't through any errors but it gets stuck in the kraken_server process. Checking the logs I can see how the client can't connect to the server, waits and tries to connect again in an infinite loop.
Trying to see what was going wrong I try to run the server kraken2-server directly which result in an error. I opened a separate issue epi2me-labs/kraken2-server#1
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
Docker
Workflow Version
2.0.7
Relevant log output
Connecting to server: localhost:8080.
Classifying sequence stream.
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...
Server is not ready: failed to connect to all addresses
Waiting 10s...Workflow fails when using test data
Hi,
I tried to call the pipeline using the testdata:
nextflow run main.nf --fastq test_data/reads.fastq.gz
This is the error I get:
* --sources: expected type: JSONObject, found: String ([TARGLOCI:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-ribosomal-survey/targeted_loci/ncbi_targeted_loci_16s_18s.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-ribosomal-survey/targeted_loci/ncbi_targeted_loci_16s_18s.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-ribosomal-survey/targeted_loci/ncbi_targeted_loci_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-ribosomal-survey/targeted_loci/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-ribosomal-survey/targeted_loci/ref2taxid.targloci.tsv, taxonomy:https://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz]])
Implement memory-mapping kraken2 option
Thanks for the --max_len flag. I will try it this evening. I tried the workflow with the test data on my other computer (Ubuntu 18.04). The default database works fine when using v2.0.1 and . However, the k2_pluspfp database fails with the following error:
[97/0c13a6] NOTE: Process kraken_pipeline:kraken2_client (1) terminated with an error exit status (74) -- Execution is retried (1)
[13/2d19cb] NOTE: Process kraken_pipeline:kraken2_client (2) terminated with an error exit status (74) -- Execution is retried (1)
[87/9b2678] NOTE: Process kraken_pipeline:kraken2_client (3) terminated with an error exit status (74) -- Execution is retried (1)
[72/37b781] NOTE: Process kraken_pipeline:kraken2_client (4) terminated with an error exit status (74) -- Execution is retried (1)
Error executing process > 'kraken_pipeline:kraken2_client (1)'
Caused by:
Process kraken_pipeline:kraken2_client (1) terminated with an error exit status (74)
Command executed:
kraken2_client --port 8080 --sequence "barcode02.4.fastq.gz" > "barcode02.kraken2.assignments.tsv"
kraken2_client --port 8080 --report --sequence "barcode02.4.fastq.gz" > "out.txt"
tail -n +2 "out.txt" > "tmp.txt"
head -n -6 "tmp.txt" > "barcode02.kraken2_report.txt"
The kraken_server was not running during this workflow. .command.log for the kraken_server process gave the following error:
Failed attempt to allocate 138245098056bytes;
you may not have enough free memory to load this database.
If your computer has enough RAM, perhaps reducing memory usage from
other programs could help you load this database?
kraken2_server: unable to allocate hash table memory
Is there a way to avoid loading the database into RAM as I only have 126 available RAM on this computer? Can --memory_mapping flag from kraken2 be implemented?
I also was able to verify that v2.0.3 behaves the same (one exception) as v2.0.1. It completes with the default database and fails (runs out of memory) with the k2_pluspfp database. The issues I was having with v2.0.3 are a result of store_dir. If this folder exists, a new database is not used (even when a different one is specified) for the workflow. So if I delete store_dir and run the workflow with the default database everything is fine. If I then run again with k2_pluspfp database, without deleting store_dir, the workflow completes but the database in store_dir has not changed. If I do the same but start with k2_pluspfp database both will fail because the k2_pluspfp database is used for both.
Thanks and I will try the --max-len flag tonight.
Scott
Originally posted by @jagos01 in #22 (comment)
Pinging home
What happened?
Not a bug report but just a note that it was a bit of an unpleasant surprise to find out that this workflow sends home data by default. I feel like that should be mentioned in the readme, as well as the disable_ping parameter.
Operating System
Windows 10
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - Execution Profile
No response
Workflow Version
master
Relevant log output
-nextflow run on the cloned wf-metagenomics repository on the local system gives an error
Hi,
I am running nextflow on a server that does not allow communication with github repositories over internet. So, I cloned the wf-metagenomics repository locally. I ran the following command: nexflow run wf-metagenomics --help
I got the following error:
`N E X T F L O W ~ version 19.01.0
Launching wf-metagenomics/main.nf [sick_heyrovsky] - revision: 710763a58e
ERROR ~ The current scope already contains a variable of the name it
@ line 421, column 33.
reads.stats.flatMap { it -> [ it[1] ] }.collect(),
^
_nf_script_9e204e56: 422: The current scope already contains a variable of the name it
@ line 422, column 30.
lineages.flatMap { it -> [ it[1] ] }.collect(),
^
_nf_script_9e204e56: 430: The current scope already contains a variable of the name it
@ line 430, column 30.
chan.flatMap { it -> [ it[1] ] }},
^
3 errors
`
I see that these lines are in the main.nf file. I am super new to nextflow and epi2me. Never worked with groovy. Could you please guide me towards figuring out this bug?
Update: I just tried running on a different server that has access to internet through nextflow run epi2me-labs/wf-metagenomics --help. It still gives the same error.
Regards,
D
[Bug]: Pipeline seems to get stuck at kraken_server/kraken_client
What happened?
Hello,
I tried to run the pipeline using the command line multiple times on my local linux pc. But it seems like it gets stuck at the kraken_server/kraken_client step. The demo data runs without a problem. For my data I try to use the PlusPF-8 database. The operating system I am using is debian 11 bullseye.
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
Singularity
Workflow Version
wf-metagenomics v2.2.1-g5344ddc
Relevant log output
N E X T F L O W ~ version 22.04.5
Launching `https://github.com/epi2me-labs/wf-metagenomics` [angry_lamarr] DSL2 - revision: 5344ddcd2a [master]
WARN: NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASE
|||||||||| _____ ____ ___ ____ __ __ _____ _ _
|||||||||| | ____| _ \_ _|___ \| \/ | ____| | | __ _| |__ ___
||||| | _| | |_) | | __) | |\/| | _| _____| |/ _` | '_ \/ __|
||||| | |___| __/| | / __/| | | | |__|_____| | (_| | |_) \__ \
|||||||||| |_____|_| |___|_____|_| |_|_____| |_|\__,_|_.__/|___/
|||||||||| wf-metagenomics v2.2.1-g5344ddc
--------------------------------------------------------------------------------
Core Nextflow options
revision : master
runName : angry_lamarr
containerEngine: singularity
launchDir : /home/xxx/nanopore/metagenomics
workDir : /home/xxx/nanopore/metagenomics/work
projectDir : /home/xxx/.nextflow/assets/epi2me-labs/wf-metagenomics
userName : xxx
profile : singularity
configFiles : /home/xxx/.nextflow/assets/epi2me-labs/wf-metagenomics/nextflow.config
Input Options
fastq : fastq
Reference Options
database_set : PlusPF-8
database_sets : [ncbi_16s_18s:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ref2taxid.targloci.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip], ncbi_16s_18s_28s_ITS:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ref2taxid.ncbi_16s_18s_28s_ITS.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip], PlusPF-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspf_08gb_20230314.tar.gz, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/new_taxdump_2023-03-01.zip], PlusPFP-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspfp_08gb_20230314.tar.gz, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/new_taxdump_2023-03-01.zip]]
Advanced Options
threads : 8
server_threads : 8
kraken_clients : 8
Other parameters
process_label : wfmetagenomics
!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-metagenomics for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
--------------------------------------------------------------------------------
This is epi2me-labs/wf-metagenomics v2.2.1-g5344ddc.
--------------------------------------------------------------------------------
Checking inputs.
Checking fastq input.
executor > local (8)
[2a/cd471e] process > fastcat (1) [100%] 1 of 1 ✔
[skipped ] process > kraken_pipeline:unpackTaxonomy [100%] 1 of 1, stored: 1 ✔
[94/fe0e6f] process > kraken_pipeline:unpackDatabase [100%] 1 of 1 ✔
[fb/d4841d] process > kraken_pipeline:determine_bracken_length [100%] 1 of 1 ✔
[- ] process > kraken_pipeline:kraken_server [ 0%] 0 of 1
[01/b09766] process > kraken_pipeline:kraken2_client (1) [ 0%] 0 of 1
[- ] process > kraken_pipeline:progressive_stats -
[- ] process > kraken_pipeline:progressive_kraken_reports -
[- ] process > kraken_pipeline:progressive_bracken -
[73/aab753] process > kraken_pipeline:getVersions [100%] 1 of 1 ✔
[78/2df5fd] process > kraken_pipeline:getParams [100%] 1 of 1 ✔
[- ] process > kraken_pipeline:makeReport -
[c3/420353] process > kraken_pipeline:output (2) [100%] 2 of 2
[- ] process > kraken_pipeline:stop_kraken_server -[ETC]: Implanting BLAST instead minimap2 in wf-metagenomics pipeline.
What happened?
Hi.
I'm trying to replace minimap2 to BLAST instead of minimap2 in the wf-metagenomics pipeline.
I looked through the whole code(Started project one day before.), but It's difficult to understand quickly;especially the starting point to implant blast tool.
I need recommendations about specific point of code to revise.
I know the question is too naive and ambiguous. So, I expect to get detailed from an ambiguous answer.
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
v2.2.1-g5344ddc
Workflow Execution - CLI Execution Profile
Docker
Workflow Version
23.04.1
Relevant log output
No output. The purpose of this issue is for revision of the code[Bug]: No such variable: recursion
What happened?
Hello,
I'm trying to run the metagenomics workflow, but I'm running into some issues.
When I try to run it I get this error:
No such variable: recursion
--check script '/usr/lib/ont-nextflow/assets/epi2me-labs/wf-metagenomics/main.nf' at line: 10
Do any of you have an idea how to fix this? I tried running another workflow and it worked.
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
nextflow run epi2me-labs/wf-metagenomics --version
Workflow Execution - CLI Execution Profile
Docker
Workflow Version
2.06
Relevant log output
Dec-13 12:22:29.421 [main] DEBUG nextflow.cli.Launcher - $> nextflow run epi2me-labs/wf-metagenomics --version
Dec-13 12:22:29.477 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 21.10.0
Dec-13 12:22:30.200 [main] DEBUG nextflow.scm.AssetManager - Git config: /usr/lib/ont-nextflow/assets/epi2me-labs/wf-metagenomics/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-metagenomics.git
Dec-13 12:22:30.213 [main] DEBUG nextflow.scm.AssetManager - Git config: /usr/lib/ont-nextflow/assets/epi2me-labs/wf-metagenomics/.git/config; branch: master; remote: origin; url: https://github.com/epi2me-labs/wf-metagenomics.git
Dec-13 12:22:30.281 [main] INFO nextflow.cli.CmdRun - Launching `epi2me-labs/wf-metagenomics` [compassionate_watson] - revision: f5d1e16131 [master]
Dec-13 12:22:30.751 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /usr/lib/ont-nextflow/assets/epi2me-labs/wf-metagenomics/nextflow.config
Dec-13 12:22:30.752 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /usr/lib/ont-nextflow/assets/epi2me-labs/wf-metagenomics/nextflow.config
Dec-13 12:22:30.758 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Dec-13 12:22:30.890 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; plugins-dir=/usr/lib/ont-nextflow/plugins
Dec-13 12:22:30.891 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Dec-13 12:22:30.899 [main] INFO org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Dec-13 12:22:30.900 [main] INFO org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Dec-13 12:22:30.903 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Dec-13 12:22:30.911 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Dec-13 12:22:30.956 [main] DEBUG nextflow.Session - Session uuid: 31dc2dfe-ad0c-490a-ac15-54d9e025947d
Dec-13 12:22:30.956 [main] DEBUG nextflow.Session - Run name: compassionate_watson
Dec-13 12:22:30.956 [main] DEBUG nextflow.Session - Executor pool size: 16
Dec-13 12:22:30.979 [main] DEBUG nextflow.cli.CmdRun -
Version: 21.10.0 build 5641
Created: 11-11-2021 18:37 UTC (18:37 BST)
System: Linux 5.15.0-56-generic
Runtime: Groovy 3.0.9 on OpenJDK 64-Bit Server VM 11.0.17+8-post-Ubuntu-1ubuntu220.04
Encoding: UTF-8 (UTF-8)
Process: 47086@GXB03790 [127.0.0.1]
CPUs: 16 - Mem: 62.7 GB (54.6 GB) - Swap: 8 GB (8 GB)
Dec-13 12:22:30.989 [main] DEBUG nextflow.Session - Work-dir: /home/grid/work [ext2/ext3]
Dec-13 12:22:31.038 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[GoogleLifeSciencesExecutor, AwsBatchExecutor, IgExecutor]
Dec-13 12:22:31.049 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Dec-13 12:22:31.064 [main] DEBUG nextflow.Session - Observer factory: TowerFactory
Dec-13 12:22:31.128 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 17; maxThreads: 1000
Dec-13 12:22:31.189 [main] DEBUG nextflow.Session - Session start invoked
Dec-13 12:22:31.192 [main] DEBUG nextflow.trace.TraceFileObserver - Flow starting -- trace file: /home/grid/output/execution/trace.txt
Dec-13 12:22:31.201 [main] DEBUG nextflow.Session - Using default localLib path: /usr/lib/ont-nextflow/assets/epi2me-labs/wf-metagenomics/lib
Dec-13 12:22:31.203 [main] DEBUG nextflow.Session - Adding to the classpath library: /usr/lib/ont-nextflow/assets/epi2me-labs/wf-metagenomics/lib
Dec-13 12:22:31.204 [main] DEBUG nextflow.Session - Adding to the classpath library: /usr/lib/ont-nextflow/assets/epi2me-labs/wf-metagenomics/lib/nfcore_external_java_deps.jar
Dec-13 12:22:32.186 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Dec-13 12:22:33.485 [main] DEBUG nextflow.Session - Session aborted -- Cause: No such property: recursion for class: nextflow.NextflowMeta
Possible solutions: version
Dec-13 12:22:33.496 [main] ERROR nextflow.cli.Launcher - @unknown
groovy.lang.MissingPropertyException: No such property: recursion for class: nextflow.NextflowMeta
Possible solutions: version
at groovy.lang.MetaClassImpl.invokeStaticMissingProperty(MetaClassImpl.java:1019)
at groovy.lang.MetaClassImpl.setProperty(MetaClassImpl.java:2862)
at groovy.lang.MetaClassImpl.setProperty(MetaClassImpl.java:3854)
at org.codehaus.groovy.runtime.InvokerHelper.setProperty(InvokerHelper.java:219)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.setProperty(ScriptBytecodeAdapter.java:496)
at nextflow.NextflowMeta$Preview.propertyMissing(NextflowMeta.groovy)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at groovy.lang.MetaClassImpl.invokeMissingProperty(MetaClassImpl.java:883)
at groovy.lang.MetaClassImpl.setProperty(MetaClassImpl.java:2864)
at groovy.lang.MetaClassImpl.setProperty(MetaClassImpl.java:3854)
at groovy.lang.GroovyObject.setProperty(GroovyObject.java:61)
at org.codehaus.groovy.runtime.InvokerHelper.setProperty(InvokerHelper.java:217)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.setProperty(ScriptBytecodeAdapter.java:496)
at Script_f0844270.runScript(Script_f0844270:10)
at nextflow.script.BaseScript.runDsl2(BaseScript.groovy:169)
at nextflow.script.BaseScript.run(BaseScript.groovy:200)
at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:221)
at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:212)
at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:120)
at nextflow.cli.CmdRun.run(CmdRun.groovy:309)
at nextflow.cli.Launcher.run(Launcher.groovy:480)
at nextflow.cli.Launcher.main(Launcher.groovy:639)[Bug]: Error executing process > 'minimap_pipeline:minimap (1)
What happened?
Hey there,
I’ve seen a similar error in your problems, but it doesn’t really match my personal documents.
I just use your metagenomics workflow for my analyses and I have some troubles. I use my own database (Silva138.1) and my own SeqId2taxid, with this I have a problem with the minimap pipelines.
I leave you my parameters and my analysis messages so that you can direct me.
Than you for you're help and if you have any questions so that I can tell you more if what I am doing so that you can help me do not hesitate.
Sincerelly
Operating System
ubuntu 20.04
Workflow Execution
EPI2ME Labs desktop application
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
None
Workflow Version
wf-metagenomics v2.2.1
Relevant log output
{
fastq: /media/stage/CL1/Stage/GD Biotech/data/Barcode01,
classifier: minimap2,
analyse_unclassified: true,
database_set: ncbi_16s_18s,
store_dir: store_dir,
reference: /media/stage/CL1/Stage/GD Biotech/Database/silva_138.fna,
bracken_level: S,
port: 8080,
host: localhost,
out_dir: /home/stage/epi2melabs/instances/wf-metagenomics_18f085b8-5883-4bb2-a686-3870d380eb3d/output,
min_len: 200,
max_len: 2000,
threads: 4,
server_threads: 8,
kraken_clients: 2,
wf: {
agent: epi2melabs/5.0.2
}
}
runName : Silva138_minimap2
containerEngine : docker
launchDir : /home/stage/epi2melabs/instances/wf-metagenomics_18f085b8-5883-4bb2-a686-3870d380eb3d
workDir : /home/stage/epi2melabs/instances/wf-metagenomics_18f085b8-5883-4bb2-a686-3870d380eb3d/work
projectDir : /home/stage/epi2melabs/workflows/epi2me-labs/wf-metagenomics
userName : stage
profile : standard
configFiles : /home/stage/epi2melabs/workflows/epi2me-labs/wf-metagenomics/nextflow.config
Input Options
fastq : /media/stage/CL1/Stage/GD Biotech/data/Barcode01
classifier : minimap2
analyse_unclassified: true
Reference Options
reference : /media/stage/CL1/Stage/GD Biotech/Database/silva_138.fna
database_sets : [ncbi_16s_18s:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ref2taxid.targloci.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip], ncbi_16s_18s_28s_ITS:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ref2taxid.ncbi_16s_18s_28s_ITS.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip], PlusPF-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspf_08gb_20230314.tar.gz, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/new_taxdump_2023-03-01.zip], PlusPFP-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspfp_08gb_20230314.tar.gz, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/new_taxdump_2023-03-01.zip]]
Output Options
out_dir : /home/stage/epi2melabs/instances/wf-metagenomics_18f085b8-5883-4bb2-a686-3870d380eb3d/output
Advanced Options
min_len : 200
max_len : 2000
threads : 4
server_threads : 8
Other parameters
process_label : wfmetagenomics
!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-metagenomics for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
--------------------------------------------------------------------------------
This is epi2me-labs/wf-metagenomics v2.2.1.
--------------------------------------------------------------------------------
Checking inputs.
Checking custom reference exists
Checking custom reference index exists
Checking fastq input.
[41/e1c32c] Submitted process > minimap_pipeline:getVersions
[4f/e218b7] Submitted process > minimap_pipeline:getParams
[5a/e6005f] Submitted process > fastcat (1)
[c5/aa803e] Submitted process > minimap_pipeline:output (1)
Staging foreign file: https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip
[7e/e55451] Submitted process > minimap_pipeline:output (2)
[de/6ea1b1] Submitted process > minimap_pipeline:unpackTaxonomy
[3e/2ab6d4] Submitted process > minimap_pipeline:minimap (1)
ERROR ~ Error executing process > 'minimap_pipeline:minimap (1)'
Caused by:
Process `minimap_pipeline:minimap (1)` terminated with an error exit status (1)
Command executed:
minimap2 -t "4" -ax map-ont "silva_138.fna" "seqs.fastq.gz" | samtools view -h -F 2304 - | workflow-glue format_minimap2 - -o "Barcode01.minimap2.assignments.tsv" -r "ref2taxid.targloci.tsv" | samtools sort -o "Barcode01.bam" -
samtools index "Barcode01.bam"
awk -F '\t' '{print $3}' "Barcode01.minimap2.assignments.tsv" > taxids.tmp
taxonkit --data-dir "taxonomy_dir" lineage -R taxids.tmp | workflow-glue aggregate_lineages -p "Barcode01.minimap2"
file1=`cat *.json`
echo "{"'"Barcode01"'": "$file1"}" >> temp
cp "temp" "Barcode01.json"
Command exit status:
1
Command output:
(empty)
Command error:
[M::mm_idx_gen::18.615*1.47] collected minimizers
[12:48:01 - workflow_glue] Starting entrypoint.
[M::mm_idx_gen::22.608*1.91] sorted minimizers
[M::main::22.953*1.89] loaded/built the index for 510508 target sequence(s)
[M::mm_mapopt_update::23.151*1.88] mid_occ = 11257
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 510508
[M::mm_idx_stat::23.288*1.88] distinct minimizers: 7252591 (61.92% are singletons); average occurrences: 18.622; average spacing: 5.534; total length: 747391099
Traceback (most recent call last):
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3652, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'JN578465.1.1478'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/stage/epi2melabs/workflows/epi2me-labs/wf-metagenomics/bin/workflow-glue", line 7, in <module>
cli()
File "/home/stage/epi2melabs/workflows/epi2me-labs/wf-metagenomics/bin/workflow_glue/__init__.py", line 62, in cli
args.func(args)
File "/home/stage/epi2melabs/workflows/epi2me-labs/wf-metagenomics/bin/workflow_glue/format_minimap2.py", line 29, in main
taxid = ref2taxid_df.at[aln.reference_name, 'taxid']
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/indexing.py", line 2412, in __getitem__
return super().__getitem__(key)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/indexing.py", line 2364, in __getitem__
return self.obj._get_value(*key, takeable=self._takeable)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/frame.py", line 3887, in _get_value
row = self.index.get_loc(index)
File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3654, in get_loc
raise KeyError(key) from err
KeyError: 'JN578465.1.1478'
Work dir:
/home/stage/epi2melabs/instances/wf-metagenomics_18f085b8-5883-4bb2-a686-3870d380eb3d/work/3e/2ab6d46c70e85acb6475e05f183388
Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
-- Check '/home/stage/epi2melabs/instances/wf-metagenomics_18f085b8-5883-4bb2-a686-3870d380eb3d/nextflow.log' file for details[Bug]: Kraken Pipeline Make Report Terminated with Error
What happened?
The metagenomics workflow will run for a period of time and start generating the html to visualize the kraken2 results and then stop with error without analyzing all of the data. Sometimes it will error out immediately and sometime it will error out 1/2 way through.
Operating System
ubuntu 20.04
Workflow Execution
EPI2ME Labs desktop application
Workflow Execution - EPI2ME Labs Versions
EPI2ME Labs V4.1.2
Workflow Execution - CLI Execution Profile
Docker
Workflow Version
v2.0.8
Relevant log output
[5a/2ddc4e] Submitted process > kraken_pipeline:progressive_bracken (348)
[cb/5ec6ee] Submitted process > kraken_pipeline:output (348)
[10/b78d93] Submitted process > kraken_pipeline:makeReport (348)
[6e/ac83c4] Submitted process > kraken_pipeline:progressive_bracken (349)
[17/50a6cb] Submitted process > kraken_pipeline:output (349)
[85/a10537] Submitted process > kraken_pipeline:makeReport (349)
[58/f00d3a] Submitted process > kraken_pipeline:progressive_bracken (350)
[10/1cbfca] Submitted process > kraken_pipeline:output (350)
[44/cb9be5] Submitted process > kraken_pipeline:makeReport (350)
[1d/e44c6d] Submitted process > kraken_pipeline:progressive_bracken (351)
[aa/9608f8] Submitted process > kraken_pipeline:output (351)
Error executing process > 'kraken_pipeline:makeReport (350)'
Caused by:
Process `kraken_pipeline:makeReport (350)` terminated with an error exit status (1)
Command executed:
report.py "wf-metagenomics-report.html" --versions versions --params params.json --summaries all_stats.350 --lineages "bracken.350" --vistempl template.html
Command exit status:
1
Command output:
(empty)
Command error:
Traceback (most recent call last):
File "/home/pasteur/epi2melabs/workflows/epi2me-labs/wf-metagenomics/bin/report.py", line 187, in <module>
main()
File "/home/pasteur/epi2melabs/workflows/epi2me-labs/wf-metagenomics/bin/report.py", line 174, in main
kraken(args.summaries[0], section)
File "/home/pasteur/epi2melabs/workflows/epi2me-labs/wf-metagenomics/bin/report.py", line 47, in kraken
lbins, lcounts = list(zip(*len_hist))
ValueError: not enough values to unpack (expected 2, got 0)
Work dir:
/home/pasteur/epi2melabs/instances/wf-metagenomics_2789f280-5845-4ae4-a505-5db33d9037da/work/44/cb9be55f8c748e639aa25f2b39ff8c
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`[Bug]: Stalls at kraken_pipeline:kraken_server
What happened?
wf-metagenomics stalls at process kraken_pipeline:kraken_server
I use wf-metagenomics v2.0.1 and nextflow 22.04.1.
I do not get issues when running wf-metagenomics v1.1.4
Command
nextflow run epi2me-labs/wf-metagenomics --fastq /data/fwa_test2/fastq_mock/ --kraken2 --threads 8
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - Execution Profile
Docker
Workflow Version
v2.0.1
Relevant log output
N E X T F L O W ~ version 22.04.1
Launching `https://github.com/epi2me-labs/wf-metagenomics` [sleepy_wright] DSL2 - revision: e5dcad0be7 [master]
WARN: NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASE
Core Nextflow options
revision : master
runName : sleepy_wright
containerEngine: docker
launchDir : /data/fwa_test2
workDir : /data/fwa_test2/work
projectDir : /home/grid/.nextflow/assets/epi2me-labs/wf-metagenomics
userName : grid
profile : standard
configFiles : /home/grid/.nextflow/assets/epi2me-labs/wf-metagenomics/nextflow.config
Core options
fastq : /data/fwa_test2/fastq_mock/
sources : [ncbi_16s_18s:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ref2taxid.targloci.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz], ncbi_16s_18s_28s_ITS:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna, refindex:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna.fai, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS_kraken2.tar.gz, kmer_dist:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/database1000mers.kmer_distrib, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ref2taxid.ncbi_16s_18s_28s_ITS.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz], PlusPF-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspf_8gb_20210517.tar.gz, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz]]
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use epi2me-labs/wf-metagenomics for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
Checking inputs.
executor > local (7)
[24/850464] process > kraken_pipeline:unpackTaxonomy [100%] 1 of 1 ✔
[b3/4fa06f] process > kraken_pipeline:unpackDatabase [100%] 1 of 1 ✔
[46/7dbb0e] process > kraken_pipeline:kraken_server [ 0%] 0 of 1
[- ] process > kraken_pipeline:combineFilterFastq -
[- ] process > kraken_pipeline:progressiveStats -
[- ] process > kraken_pipeline:kraken2_client -
[- ] process > kraken_pipeline:progressive_kreports -
[- ] process > kraken_pipeline:taxon_kit -
[- ] process > kraken_pipeline:bracken -
[eb/abc3f8] process > kraken_pipeline:getVersions [100%] 1 of 1 ✔
[b9/f9f63c] process > kraken_pipeline:getParams [100%] 1 of 1 ✔
[- ] process > kraken_pipeline:makeReport -
[- ] process > kraken_pipeline:mergeclassifiedProgressive -
[- ] process > kraken_pipeline:mergeunclassifiedProgressive -
[- ] process > kraken_pipeline:catAssignmentsprogressive -
[- ] process > kraken_pipeline:stop_kraken_server -
[- ] process > kraken_pipeline:output -
[- ] process > kraken_pipeline:output_dir -
[a2/942ffb] process > output (1) [100%] 2 of 2 ✔[Bug]: The README doesn't state how the database has been built
What happened?
Species assignment can be very different depending on the database used. And as they said, "the species assignment is only as good as the database you use to match against".
There's no info much info about how the database was created or where the reference sequences were taken from. Looking at the code I can see that the database is being downloaded from ONT cloud servers, the name mentions "targeted loci" so I assume it's been built with some of the parts of the NCBI's Targeted Loci project. But it's not mentioned anywhere which parts.
Could you make that info available?
Thanks
Operating System
ubuntu 20.04
Workflow Execution
EPI2ME Labs desktop application
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
None
Workflow Version
2.0.8
Relevant log output
No logs[Bug]: Watchmode error in makeReport
Hi all,
Awesome pipeline! I'm running the latest version on my Nanopore fastq data using the following command using Nextflow version 22.10.5:
nextflow run wf-metagenomics/main.nf --fastq test --database_set "PlusPF-8" --watch_path --batch_size 1
Everything works for the standard mode but as soon as I add a new fastq file to the folder, the pipeline crashes within the report step (" ValueError: not enough values to unpack (expected 2, got 0)")
Am I missing something? Thanks for your help!
Error executing process > 'kraken_pipeline:makeReport (1)'
Caused by:
Process `kraken_pipeline:makeReport (1)` terminated with an error exit status (1)
Command executed:
report.py "wf-metagenomics-report.html" --versions versions --params params.json --summaries all_stats.1 --lineages "bracken.1" --vistempl template.html
Command exit status:
1
Command output:
(empty)
Command error:
Traceback (most recent call last):
File "2023-nanopore-metagenomics/wf-metagenomics/bin/report.py", line 187, in <module>
main()
File "2023-nanopore-metagenomics/wf-metagenomics/bin/report.py", line 174, in main
kraken(args.summaries[0], section)
File "2023-nanopore-metagenomics/wf-metagenomics/bin/report.py", line 47, in kraken
lbins, lcounts = list(zip(*len_hist))
ValueError: not enough values to unpack (expected 2, got 0)
Work dir:
2023-nanopore-metagenomics/work/18/3e70a6c69ae8aba691b68373851170
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
Operating System
ubuntu 18.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
None
Workflow Version
v2.0.8
Relevant log output
[skipped ] process > kraken_pipeline:unpackTaxonomy [100%] 1 of 1, stored: 1 ✔
[skipped ] process > kraken_pipeline:unpackDatabase [100%] 1 of 1, stored: 1 ✔
[e9/e1bb00] process > kraken_pipeline:determine_bracken_length [100%] 1 of 1 ✔
[89/e934d0] process > kraken_pipeline:kraken_server [ 0%] 0 of 1
[bf/54d610] process > kraken_pipeline:rebatchFastq (4) [100%] 4 of 4
[bb/d90787] process > kraken_pipeline:kraken2_client (7) [ 0%] 7 of 4000
[9b/e93d0e] process > kraken_pipeline:progressive_stats (7) [ 85%] 6 of 7
[aa/9341be] process > kraken_pipeline:progressive_kraken_reports (7) [ 85%] 6 of 7
executor > local (37)
[skipped ] process > kraken_pipeline:unpackTaxonomy [100%] 1 of 1, stored: 1 ✔
[skipped ] process > kraken_pipeline:unpackDatabase [100%] 1 of 1, stored: 1 ✔
[e9/e1bb00] process > kraken_pipeline:determine_bracken_length [100%] 1 of 1 ✔
[89/e934d0] process > kraken_pipeline:kraken_server [ 0%] 0 of 1
[bf/54d610] process > kraken_pipeline:rebatchFastq (4) [100%] 4 of 4
[bb/d90787] process > kraken_pipeline:kraken2_client (7) [ 0%] 7 of 4000
[9b/e93d0e] process > kraken_pipeline:progressive_stats (7) [ 85%] 6 of 7
[aa/9341be] process > kraken_pipeline:progressive_kraken_reports (7) [ 85%] 6 of 7
[a4/61917d] process > kraken_pipeline:progressive_bracken (3) [ 66%] 2 of 3
executor > local (37)
[skipped ] process > kraken_pipeline:unpackTaxonomy [100%] 1 of 1, stored: 1 ✔
[skipped ] process > kraken_pipeline:unpackDatabase [100%] 1 of 1, stored: 1 ✔
[e9/e1bb00] process > kraken_pipeline:determine_bracken_length [100%] 1 of 1 ✔
[89/e934d0] process > kraken_pipeline:kraken_server [ 0%] 0 of 1
[bf/54d610] process > kraken_pipeline:rebatchFastq (4) [100%] 4 of 4
[bb/d90787] process > kraken_pipeline:kraken2_client (7) [ 0%] 7 of 4000
[9b/e93d0e] process > kraken_pipeline:progressive_stats (7) [ 85%] 6 of 7
[aa/9341be] process > kraken_pipeline:progressive_kraken_reports (7) [ 85%] 6 of 7
[a4/61917d] process > kraken_pipeline:progressive_bracken (3) [ 66%] 2 of 3
[a8/e98d8a] process > kraken_pipeline:getVersions [100%] 1 of 1 ✔
[a7/fe03de] process > kraken_pipeline:getParams [100%] 1 of 1 ✔
[18/3e70a6] process > kraken_pipeline:makeReport (1) [ 50%] 1 of 2, failed: 1
[5a/e349bd] process > kraken_pipeline:output (2) [100%] 2 of 2
[- ] process > kraken_pipeline:stop_kraken_server -
Error executing process > 'kraken_pipeline:makeReport (1)'
Caused by:
Process `kraken_pipeline:makeReport (1)` terminated with an error exit status (1)
Command executed:
report.py "wf-metagenomics-report.html" --versions versions --params params.json --summaries all_stats.1 --lineages "bracken.1" --vistempl template.html
Command exit status:
1
Command output:
(empty)
Command error:
Traceback (most recent call last):
File "2023-nanopore-metagenomics/wf-metagenomics/bin/report.py", line 187, in <module>
main()
File "2023-nanopore-metagenomics/wf-metagenomics/bin/report.py", line 174, in main
kraken(args.summaries[0], section)
File "2023-nanopore-metagenomics/wf-metagenomics/bin/report.py", line 47, in kraken
lbins, lcounts = list(zip(*len_hist))
ValueError: not enough values to unpack (expected 2, got 0)
Work dir:
work/18/3e70a6c69ae8aba691b68373851170
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`Barcode02 in test data produces a lot of unclassified reads
What happened?
Command:
nextflow run epi2me-labs/wf-metagenomics --fastq /home/scott/.nextflow/assets/epi2me-labs/wf-metagenomics/test_data --kraken2 --out_dir /home/scott/output_55
The workflow finished but Barcode02 produced a large number of unclassified reads. The same results are produced when I run the test data using a local copy of the k2_pluspfp database (129 GB).
When I run a subset of my own data, the reads are largely unclassified. Any ideas on how to resolve this are appreciated.
Thanks,
Scott
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - Execution Profile
Docker
Workflow Version
af0af1b [master]
Relevant log output
-[Bug]: /bin/bash: .command.run: No such file or directory
What happened?
Hi, I am trying to use the wf-metagenomics analysis with my data and the PlusPF-8 database. There is no problem when I try the demo data. However, when I use my own data, the execution stops at the fastcat step and gives me the following error message: "/bin/bash: .command.run: No such file or directory". I also wonder about the path, as the demo data used "/mnt/c/Users//epi2melabs/instances/".
Operating System
Windows 10
Workflow Execution
EPI2ME Labs desktop application
Workflow Execution - EPI2ME Labs Versions
EPI2ME V5.0.2
Workflow Execution - CLI Execution Profile
Docker
Workflow Version
Version: v2.2.1 | 5344ddc
Relevant log output
This is epi2me-labs/wf-metagenomics v2.2.1.
--------------------------------------------------------------------------------
Checking inputs.
Checking fastq input.
[8f/68926d] Submitted process > kraken_pipeline:getVersions
[11/47913c] Submitted process > fastcat (1)
[0d/bc5a51] Submitted process > kraken_pipeline:getParams
ERROR ~ Error executing process > 'fastcat (1)'
Caused by:
Process `fastcat (1)` terminated with an error exit status (127)
Command executed:
mkdir fastcat_stats
fastcat -s fastq_pass -r fastcat_stats/per-read-stats.tsv -f fastcat_stats/per-file-stats.tsv fastq_pass | bgzip -@ 2 > seqs.fastq.gz
Command exit status:
127
Command output:
(empty)
Command error:
/bin/bash: .command.run: No such file or directory
Work dir:
/mnt/wsl/docker-desktop-bind-mounts/Ubuntu/fdeace0c97c37e435605c4867be6a665446a517252e505ab6a2b1207e75ae9a7/wsl/docker-desktop-bind-mounts/Ubuntu/b79cc489979f44b51ed29b5d09aa9a5c6eff0cbfeecccdd851577e524db4634c/instances/wf-metagenomics_cb6bc4cf-159f-4dca-b944-23da743ce4d0/work/11/47913cfa292416a6653804f81b2af1
Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
-- Check '/mnt/wsl/docker-desktop-bind-mounts/Ubuntu/fdeace0c97c37e435605c4867be6a665446a517252e505ab6a2b1207e75ae9a7/wsl/docker-desktop-bind-mounts/Ubuntu/b79cc489979f44b51ed29b5d09aa9a5c6eff0cbfeecccdd851577e524db4634c/instances/wf-metagenomics_cb6bc4cf-159f-4dca-b944-23da743ce4d0/nextflow.log' file for details
WARN: Killing running tasks (1)
WARN: Graphviz is required to render the execution DAG in the given format -- See http://www.graphviz.org for more info.I just had the `--threads` error too. I used to run the workflow with `--threads 24`. Today I updated to the newest version and its not working anymore.
I just had the `--threads` error too. I used to run the workflow with `--threads 24`. Today I updated to the newest version and its not working anymore.
I get this error:
Error executing process > 'kraken_pipeline:kraken_server'
Caused by:
Process requirement exceeds available CPUs -- req: 24; avail: 8
Command executed:
# we add one to requests to allow for stop signal
kraken2_server --max-requests 24 --port 8080 --db ./database_dir/
Command exit status:
-
Command output:
(empty)
Work dir:
/home/DNAdminseq/dnasqdatadrive/Projects/dnaseq/metagenomics/themis/wf-metagenomics/test/work/47/9e8d5b49c0e805e6118350506b106d
Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
I also have a bit unrelated question. Everytime I try to rerun the same sample (for example if it got stuck due to an error) I always need to delete the trace.txt file in the output/execution folder. Is this how its supposed to be or is there a workaround?
Originally posted by @eparisis in #27 (comment)
High runtime when executing workflow on metagenomic data, especially in the centrifuge step
Hello community,
I recently performed a metagenomic analysis on sequencing data generated from a zymomock community and tested this nextflow workflow.
Unfortunately, I had the issue that the complete execution of the workflow required exceptional high runtime compared to the corresponding EPI2ME Labs Notebook 'Metagenomic classification tutorial'. In particular, it seems that the centrifuge step is the reason for this behaviour.
This is the command, which I have used to run the workflow:
nextflow run epi2me-labs/wf-metagenomics \
-w output/workspace \
-profile standard \
--fastq ../fastq_pass \
--db_path db_store/zymo \
--db_prefix zymo \
--out_dir output \
--wfversion latest \
--threads 16
I used the centrifuge database files from the test_data/ directory of this repo, which can be found here.
About my environment:
Nextflow Version: 21.04.3 build 5560
Created: 21-07-2021 15:09 UTC (17:09 CEST)
System: Linux 5.4.0-84-generic
Runtime: Groovy 3.0.7 on OpenJDK 64-Bit Server VM 11.0.11+9-Ubuntu-0ubuntu2.18.04
Encoding: UTF-8 (UTF-8)
CPUs: 32 - Mem: 125.8 GB (14.6 GB) - Swap: 2 GB (1.4 GB)
Launching `epi2me-labs/wf-metagenomics` [stoic_edison] - revision: ef8303bb01 [master]
The workflow terminated successfully after 3 h and 29 min:
executor > local (11)
[82/24dd36] process > pipeline:fastcat (1) [100%] 1 of 1 ✔
[4d/c2894e] process > pipeline:centrifuge (1) [100%] 1 of 1 ✔
[09/bf6e7f] process > pipeline:generateMaster (1) [100%] 1 of 1 ✔
[86/1ddea2] process > pipeline:splitByMaster (1) [100%] 1 of 1 ✔
[a1/b957be] process > output (7) [100%] 7 of 7 ✔
Completed at: 08-Oct-2021 18:34:51
Duration : 3h 29m
CPU hours : 3.3
Succeeded : 11
According to the 'Process execution timeline' that nextlfow automatically generates in the output/execution/ directory for every run, the most time demanding step was the execution of centrifuge (with almost 3 h):
In contrast, the EPI2ME Labs Notebook only required about 6 min for the centrifuge step on the same dataset with equal centrifuge database files when running with 16 threads. Therefore, it seems like this workflow is facing some critical performance issues.
I have also checked the number of threads generated for the execution of centrifuge during the run and found 16 (like specified in the command using the --threads flag). However, the average CPU usage of centrifuge for each core was very low (< 10%). When running the EPI2ME Labs Notebook, the CPU usage was at nearly 100 % for the centrifuge step.
Until now, I was not able to identify the problem. Do you have any suggestions?
Barcode02 in test data produces a lot of unclassified reads
What happened?
Command:
nextflow run epi2me-labs/wf-metagenomics --fastq /home/scott/.nextflow/assets/epi2me-labs/wf-metagenomics/test_data --kraken2 --out_dir /home/scott/output_55
The workflow finished but Barcode02 produced a large number of unclassified reads. The same results are produced when I run the test data using a local copy of the k2_pluspfp database (129 GB).
When I run a subset of my own data, the reads are largely unclassified. Any ideas on how to resolve this are appreciated.
Thanks,
Scott
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - Execution Profile
Docker
Workflow Version
af0af1b [master]
Relevant log output
-[Bug]: Error: unknown lineage Lu
What happened?
We've just hit the error below. After some diagnosing we've noticed that aggregate_lineages_bracken.py is hitting a line in lineages.txt that contains only one string: Lu. This produces an error when the function update_or_create_count() (see below) tries to unpack it because it's expecting three tab separated fields instead of one string
https://github.com/epi2me-labs/wf-metagenomics/blob/master/bin/aggregate_lineages.py#L47
We also noticed that lineages.txt is created by taxonkit using the database ncbi_16s_18s_28s_ITS_kraken2.tar.gz. So we believe that the culprit is a special character somewhere in the taxo.t2k file of that database. Specifically, we think in our case the problem is an organism called " 'Massilia aquatica' Lu et al. 2020 ". We think the single quotation marks are the issue.
Operating System
ubuntu 20.04
Workflow Execution
EPI2ME Labs desktop application
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - Execution Profile
No response
Workflow Version
v2.0.1
Relevant log output
Error executing process > 'kraken_pipeline:bracken (1)'
Caused by:
Process `kraken_pipeline:bracken (1)` terminated with an error exit status (1)
Command executed:
run_bracken.py "database_dir" "reports.1/8_CB541_substrate_Qiagen_40.kreport.txt" "1000" "S" "8_CB541_substrate_Qiagen_40.bracken_report.txt"
mv "reports.1/8_CB541_substrate_Qiagen_40.kreport_bracken_species.txt" .
awk '{ print $3,$7}' "8_CB541_substrate_Qiagen_40.bracken_report.txt" | awk 'NR!=1 {print}' > taxacounts.txt
awk '{print $3}' "8_CB541_substrate_Qiagen_40.bracken_report.txt" | awk 'NR!=1 {print}' > taxa.txt
taxonkit --data-dir taxonomy_dir lineage -R taxa.txt > lineages.txt
aggregate_lineages_bracken.py -i "lineages.txt" -b "taxacounts.txt" -p "8_CB541_substrate_Qiagen_40.kraken2"
file1=`cat *.json`
echo "{"'"8_CB541_substrate_Qiagen_40"'": "$file1"}" >> "8_CB541_substrate_Qiagen_40.1.json"
cp "8_CB541_substrate_Qiagen_40.1.json" "reports.1/8_CB541_substrate_Qiagen_40.json"
Command exit status:
1
Command output:
b' >> Checking for Valid Options...\n >> Running Bracken \n >> python src/est_abundance.py -i reports.1/8_CB541_substrate_Qiagen_40.kreport.txt -o 8_CB541_substrate_Qiagen_40.bracken_report.txt -k database_dir/database1000mers.kmer_distrib -l S -t 0\nPROGRAM START TIME: 11-11-2022 18:39:50\nBRACKEN SUMMARY (Kraken report: reports.1/8_CB541_substrate_Qiagen_40.kreport.txt)\n >>> Threshold: 0 \n >>> Number of species in sample: 203 \n\t >> Number of species with reads > threshold: 203 \n\t >> Number of species with reads < threshold: 0 \n >>> Total reads in sample: 4308\n\t >> Total reads kept at species level (reads > threshold): 751\n\t >> Total reads discarded (species reads < threshold): 0\n\t >> Reads distributed: 174\n\t >> Reads not distributed (eg. no species above threshold): 328\n\t >> Unclassified reads: 3056\nBRACKEN OUTPUT PRODUCED: 8_CB541_substrate_Qiagen_40.bracken_report.txt\nPROGRAM END TIME: 11-11-2022 18:39:50\n Bracken complete.\n'Error: unknown lineage Lu
Command error:
b'>> Checking report file: reports.1/8_CB541_substrate_Qiagen_40.kreport.txt\n'
Work dir:
/home/concertbio/epi2melabs-data/nextflow/instances/2022-11-11-18-38_wf-metagenomics_kFhzGMeFDsQWUdWWiJrs2U/work/f0/beb156eb6a8d2c4ad046e6ced954b5
Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`[Bug]: Include ".kraken2.assignments.tsv" in the output folder.
What happened?
I was wondering if there is a way to add the .kraken2.assignments.tsv to the outputs since its very useful for downstream processing.
I'm relatively new to nextflow and I was wondering if there is a way to add the ${sample_id}.kraken2.assignments.tsv which is generated in the kraken2_client process with the publishDir function.
If I try adding to the process outputs path "${sample_id}.kraken2.assignments.tsv" and publishDir path: "${params.out_dir}", mode: 'copy', pattern: "${sample_id}.kraken2.assignments.tsv", saveAs: {name -> "kraken"}, overwrite: true to the process I get following error:
Invalid method invocation `call` with arguments: [barcode01, ../work/ee/00291fe5954f130ec83c0ce60706a9/barcode01.kraken2.report.txt, ../work/ee/00291fe5954f130ec83c0ce60706a9/barcode01.2.json, ../work/ee/00291fe5954f130ec83c0ce60706a9/barcode01.kraken2.assignments.tsv] (java.util.ArrayList) on _closure65 typeAny suggestions to make it work?
Operating System
ubuntu 20.04
Workflow Execution
Command line
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - CLI Execution Profile
Docker
Workflow Version
Latest
Relevant log output
-Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.