nextflow-io / nextflow Goto Github PK
View Code? Open in Web Editor NEWA DSL for data-driven computational pipelines
Home Page: http://nextflow.io
License: Apache License 2.0
A DSL for data-driven computational pipelines
Home Page: http://nextflow.io
License: Apache License 2.0
In the following example, the print process can't access the x
file.
process read_data {
output:
file "bla*" into read
script:
"""
echo 'Hello World' > bla.txt
"""
}
process print {
input:
file x from read
exec:
x.eachLine { str ->
println str
}
}
Nextflow installer download required dependencies on-demand by using Capsule project.
Unfortunately it does not support proxy server, thus when running behind a proxy server the installer in unable to download required dependencies.
For example:
Channel
.path('misc/sample.fa')
.splitFasta( record: [id: true, seqString: true ])
.filter { record -> record.id =~ /.*_B$/ }
.subscribe { record -> println record.seqString }
It returns
The dataflow processor experienced an unhandled exception and is about to terminate. java.lang.ClassCastException: java.util.regex.Matcher cannot be cast to java.lang.Boolean
Moreover the application hung
Recently, I have been working on a nextflow pipeline where I need to log all my output of 3 processes into a single file and I have to save the resulting files.
To attend this goal, I have build two Channels:
qc_logger.subscribe { logger(it) }
qc_copy.subscribe { copyFile(it, read_folder) }
I have to repeat the same code for every process.
I have also tried to make a long living channel, but after the first process completed the channel was soon closed ignoring the output of others process.
Is there any work around? or include support for a long living Channel?
For example
ch = file(params.in)
process foo {
input:
file x from ch
"echo ${x.toAbsolutePath()}"
}
It is expected that it resolves the path against the process working path instead it is resolved agains the main thread execution path.
I really like the integration of Github to version the pipeline files and hope nextflow to support local Git , not only Github. For example, we have local Gitlab for our internal code management. It could be better to support local Git repository. Thanks!
The documentation for moveTo
method states that:
When a file with the same name as the target already exists, it will be replaced by the new one.
However currently a FileAlreadyExistsException is thrown when the a file already exist in the target directory.
there is 50 lines in my valid file .
why I got only one file in the validUtrChannelSplit Channel?
I am confused.
params.mirfile = "/llwang-works/data/ceRNA/example_nextflow/data/mir.expr"
params.utrfile = "/llwang-works/data/ceRNA/example_nextflow/data/utr.expr"
params.chunkSize = 8
mirIo = file(params.mirfile)
utrIo = file(params.utrfile)
mirChannel = Channel.from(mirIo)
utrChannel = Channel.from(utrIo)
process validate_utr {
input:
file 'source' from utrChannel
output:
file 'valid' into validUtrChannel
"""
perl /llwang-works/data/ceRNA/example_nextflow/scripts/validate.pl source > valid
"""
}
process validate_mir {
input:
file 'source' from mirChannel
output:
file 'valid' into validMirChannel
"""
perl /llwang-works/data/ceRNA/example_nextflow/scripts/validate.pl source > valid
"""
}
validUtrChannelSplit = validUtrChannel.splitText(by: params.chunkSize)
process task_cat {
input:
file 'seq' from validUtrChannelSplit
file 'db' from validMirChannel
output:
file 'out' into corChannel
"""
cat seq > out
"""
}
Currently, if maxErrors is set to a value lesser than maxRetries, only up to maxErrrors re-submissions are attempted for a process. From a user's perspective, this is a bit confusing, especially if only maxRetries is set and maxErrors is left to its default value (1). maxErrors should not limit the number of possible resubmissions.
From outside the box, I see to alternative ways to go:
I'd like to give this a try. I am trying to install on a cluster on which I do no have sudo. I use my own java, and the path to the executable is:
/home/stephen/bin/jre1.7.0/bin/java
Somewhere in lines 174-179 of the installer script it's trying to figure out where java lives, and it gets hung up at:
Nextflow cannot start -- unable to run Java executable: /home/stephen/bin/jre1.7.0/bin/java/bin/java
Apparently it's trying to add another /bin/java
to the path. How can I fix this? Thanks.
Docker images encapsulating individual applications are generally set up with an ENTRYPOINT
instruction, which makes the image executable. In many deployments, it is considered good practice to set up images and containers in this way.
In these cases, the appropriate way to interact with the image is to pass a list of arguments to the executable to docker run
, rather than to call it with a command to run a Bash script.
It would be much easier to interact with and use executable images if this support were available.
Processes are executed even when cached results of the same process with the same inputs are available.
Consider this minimal example (or git clone https://gist.github.com/b189ef40129bf751c8f8.git
):
#!/usr/bin/env nextflow
params.query = './*.fasta'
fastaFiles = Channel.fromPath(params.query)
process getDefinitions {
input:
file 'fasta' from fastaFiles
output:
stdout defLines
"""
sleep 10
awk '/^>/ {print(substr(\$0,2))}' $fasta
"""
}
defLines.subscribe {
println it
}
The nextflow documentation indicates that
(Any) attempt to execute the process along with the same inputs will cause the process execution to be skipped
I was expecting that if the program was re-run using the same inputs, the getDefinitions process would be skipped and the cached results used in place. This doesn't seem to happen:
$ rm -r work
$ time nextflow main.nf --query example.fasta
N E X T F L O W ~ version 0.10.2
[warm up] executor > local
[f8/05f1ee] Submitted process > getDefinitions (1)
SEQUENCE_1 [Homo sapiens sapiens]
SEQUENCE_2 [Homo sapiens neanderthalensis]
SEQUENCE_3 [Mus musculus musculus]
real 0m12.070s
user 0m3.034s
sys 0m0.341s
$ time nextflow main.nf --query example.fasta
N E X T F L O W ~ version 0.10.2
[warm up] executor > local
[ad/28e06b] Submitted process > getDefinitions (1)
SEQUENCE_1 [Homo sapiens sapiens]
SEQUENCE_2 [Homo sapiens neanderthalensis]
SEQUENCE_3 [Mus musculus musculus]
real 0m12.041s
user 0m2.908s
sys 0m0.409s
I note that the hashLog values change for each indication. I had a look at the source, and it looks like when CacheHelper.hasher is given a file, it returns a Hasher that constructs a hash from:
file.exists()
)This is probably my inexperience with java showing, but I don't understand why the same file doesn't produce the same hash.
It is difficult to construct significant workflows when the whole thing needs to be re-run every time. Is this a bug, or just my mis-use?
That way this will work:
file('env://MY_INPUT_FILE')
and will load whatever the value of MY_INPUT_FILE
is set to in the environment.
When putting echo true
into my pipeline script, I get nice logging output that I can then redirect to a file. This would work even better for me if I could specify a file to direct/copy the stdout/stderr output to when using echo true
.
This way, I would just see the (much smaller) list of the executed commands when executing nextflow run
and still have the full log present if I (maybe days later) see that something went wrong with the pipeline run.
I don`t know how do I put file into channel, the file lie in a directory.
while directory is a parameter rather than a file.
such as programe "tophat ".
the command line like following:
$ tophat -o tophat_out genome_index R1.fq R2.fq
tophat_out/:
accepted_hits.bam deletions.bed junctions.bed prep_reads.info
then I want pass the file "accepted_hits.bam" into a Channel, How should I do?
hearing from you!
Hi
Running gradle test with last version I get:
groovy.lang.MissingPropertyException at CacheHelperTest.groovy:38
ERROR:
groovy.lang.MissingMethodException: No signature of method: embed.com.google.common.hash.Murmur3_128HashFunction$Murmur3_128Hasher.putString() is applicable for argument types: (java.lang.String) values: [Hola]
Possible solutions: putString(java.lang.CharSequence, java.nio.charset.Charset), toString(), toString(), chopString(), putLong(long), chopString(java.util.Map)
at nextflow.util.CacheHelperTest.test hashCode(CacheHelperTest.groovy:38)
I can solve with
def aString = Hashing.murmur3_128().newHasher().putString('Hola', Charset.defaultCharset()).hash()
But I get more errors from the following tests
Best Regards
A NullPointerException is raised when a native process is executed using the gridgain
executor
This might not be an bug, but here's a simple nf script to demonstrate. Replacing val(fastq.baseName + '.out')
with val('abcd' + '.out')
works.
#!/usr/bin/env nextflow
seqs = file('seqs/a.fastq')
process proc1 {
input:
file fastq from seqs
output:
val(fastq.baseName + '.out') into proc1
"""
echo 1
"""
}
N E X T F L O W ~ version 0.13.4-SNAPSHOT
Launching strcat.nf
ERROR ~ Not such variable: fastq
-- Check script 'strcat.nf' at line: 9 or see '.nextflow.log' file for more details
.nextflow.log:
https://gist.github.com/yesimon/fab422b9442ff83fd0c8
When two or more processes have an identical command stript and get the identical inputs the and identical hash key is assigned to different process instances.
This result in multiple processes generating the command script in the same working directory and unpredictable content of script and wrapper files.
I tried to resolve my problem described in #18 with the pattern proposed there. I'm opening another ticket here since I think this is a separate issue. The section below shows my script, the call and error message and the work directory.
Am I missing something or doing something wrong?
holtgrem@t-jbot01:~/Software/ngspipeline$ cat pipeline.nextflow
#!/usr/bin/env nextflow
import ChannelUtil
import FastQC
import PathUtil
import PipelinePaths
// Uncomment for seeing the output of th eprocesses.
echo true
params.runID = 1234
params.baseDir = "$HOME/Projects/nextflow-test"
// Create helper for generating pipeline file paths.
paths = new PipelinePaths(params.baseDir);
// Open channel for left and right files and merge it into triples, the
// first entry is the LCS of the file names that can be used as a read
// pair identifier.
readPairs = ChannelUtil.createFilePairChannel(
Channel.fromPath(paths.fastqPattern("original", "R1")),
Channel.fromPath(paths.fastqPattern("original", "R2"))
)
// Genome file.
// TODO(holtgrew): Is this really required or could we as well use the path?
genomeFile = file(params.genome)
// Duplicate the read pairs into one queue for runFastQCOriginal
// and runTrimming.
readPairsFastQCOriginal = Channel.create()
readPairsRunTrimming = Channel.create()
readPairs.separate(readPairsFastQCOriginal, readPairsRunTrimming) { x -> [ x, x ] }
// --------------------------------------------------------------------------
// Run FastQC on the original reads.
// --------------------------------------------------------------------------
process runFastQCOriginal {
module 'fastqc/0.11.2'
input:
set pairID, file(readL), file(readR) from readPairsFastQCOriginal
output:
file 'original/*.zip' into fastqcOutputOriginal
script:
"""
set -x
mkdir -p original
fastqc -o original ${readL} ${readR}
"""
}
holtgrem@t-jbot01:~/Software/ngspipeline$ nextflow run pipeline.nextflow
N E X T F L O W ~ version 0.11.3
[warm up] executor > local
[dc/2e80a7] Submitted process > runFastQCOriginal (1)
[83/572a74] Submitted process > runFastQCOriginal (2)
Error executing process > 'runFastQCOriginal (1)'
Caused by:
Missing output file(s): 'original/*.zip' expected by process: runFastQCOriginal (1)
Command executed:
set -x
mkdir -p original
fastqc -o original reads001_R1.fastq.gz reads002_R2.fastq.gz
Command exit status:
0
Command output:
+ mkdir -p original
+ fastqc -o original reads001_R1.fastq.gz reads002_R2.fastq.gz
Started analysis of reads001_R1.fastq.gz
Analysis complete for reads001_R1.fastq.gz
Started analysis of reads002_R2.fastq.gz
Analysis complete for reads002_R2.fastq.gz
Work dir:
/media/3dfdd68f-a74e-4fb8-8e7f-203343c39564/data6/home/holtgrem/Software/ngspipeline/work/dc/2e80a74320773446b671eec63f0083
Tip: view the complete command output by changing to the process work dir and entering the command: 'cat .command.out'
holtgrem@t-jbot01:~/Software/ngspipeline$ tree /media/3dfdd68f-a74e-4fb8-8e7f-203343c39564/data6/home/holtgrem/Software/ngspipeline/work/dc/2e80a74320773446b671eec63f0083
/media/3dfdd68f-a74e-4fb8-8e7f-203343c39564/data6/home/holtgrem/Software/ngspipeline/work/dc/2e80a74320773446b671eec63f0083
|-- original
| |-- reads001_R1_fastqc.html
| |-- reads001_R1_fastqc.zip
| |-- reads002_R2_fastqc.html
| `-- reads002_R2_fastqc.zip
|-- reads001_R1.fastq.gz -> /home/holtgrem/Projects/nextflow-test/fastq/original/reads001_R1.fastq.gz
`-- reads002_R2.fastq.gz -> /home/holtgrem/Projects/nextflow-test/fastq/original/reads002_R2.fastq.gz
1 directory, 6 files
GitHub public repositories have a very low rate limits. See https://developer.github.com/v3/#rate-limiting
When trying to download a repository for which the user reached the access rate limit, Nextflow reports that the repository does not exists.
Cannot find cbcrg/piper-nf pipeline -- Make sure exists a GitHub repository at https://github.com/cbcrg/piper-nf
This is confusing the user. It should provide a more detailed error message suggesting to enter its credentials.
HTTP/1.1 403 Forbidden
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Content-Encoding: gzip
Content-Security-Policy: default-src 'none'
Content-Type: application/json; charset=utf-8
Date: Tue, 09 Sep 2014 12:41:08 GMT
Server: GitHub.com
Status: 403 Forbidden
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
Transfer-Encoding: chunked
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-GitHub-Media-Type: github.v3
X-GitHub-Request-Id: 545842C2:592E:19837FF:540EF564
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1410267414
X-XSS-Protection: 1; mode=block
{
"documentation_url": "https://developer.github.com/v3/#rate-limiting",
"message": "API rate limit exceeded for 84.88.66.194. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)"
}
See the log file in the above link and the example at #41
Modify the build script in such a way to include the git commit number in version information number
a = Channel.fromPath('f/*.txt')
b = Channel.fromPath('f/*.txt')
c = a.merge(b) { one, two -> [one, two]}
process handle_paired {
echo true
input:
file c
script:
c_0 = c[0]
"""
echo $c_0
"""
}
N E X T F L O W ~ version 0.14.2
Launching integer.nf
[warm up] executor > local
Error executing process > 'handle_paired (1)'
Caused by:
No signature of method: nextflow.util.BlankSeparatedList.getAt() is applicable for argument types: (java.lang.Integer) values: [0]
Possible solutions: getAt(java.lang.String), get(int), set(int, java.lang.Object), putAt(java.lang.String, java.lang.Object), wait(), grep()
Source block:
c_0 = c[0]
"""
echo $c_0
"""
Tip: view the complete command output by changing to the process work dir and entering the command: 'cat .command.out'
-- Check script 'integer.nf' at line: 12 or see '.nextflow.log' file for more details
My cextflow.config is defined as follows:
process {
executor='sge'
queue='all.q'
}
Then I run the following:
nextflow run tutorial.nf
N E X T F L O W ~ version 0.11.2
[warm up] executor > sge
[ad/b159ca] Submitted process > splitLetters (1)
At which point the job will appear in the SGE queue, hit an open slot, then disappear (not sure if it fails or not).. and the Submitted process > splitLetters (1)
just hangs.
x = 1
process foo {
input:
val y from 'a'
output:
file "${x}_${y}" into q
"""
echo Hello
"""
}
The following exception is thrown
Nov-11 15:40:53.701 [Actor Thread 3] ERROR nextflow.processor.TaskProcessor - Error executing process > 'foo (1)'
Caused by:
x
groovy.lang.MissingPropertyException: x
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_51]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.script.BaseParam.resolveName(ParamsIn.groovy:205) ~[nextflow-0.11.0.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:368) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.script.FileOutParam$_getFilePatterns_closure1.doCall(ParamsOut.groovy:238) ~[nextflow-0.11.0.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.script.TokenGString$_resolve_closure1.doCall(ScriptTokens.groovy:130) ~[nextflow-0.11.0.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.Closure.call(Closure.java:439) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:2348) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:2318) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.dgm$63.invoke(Unknown Source) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.script.TokenGString.resolve(ScriptTokens.groovy:130) ~[nextflow-0.11.0.jar:na]
at nextflow.script.TokenGString$resolve.call(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.script.FileOutParam.getFilePatterns(ParamsOut.groovy:238) ~[nextflow-0.11.0.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:166) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:68) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.processor.TaskRun$_memoizedMethodPriv$getOutputFilesNames_closure8.doCall(TaskRun.groovy:296) ~[nextflow-0.11.0.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.Closure.call(Closure.java:439) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1379) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1372) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.dgm$149.invoke(Unknown Source) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.processor.TaskRun.memoizedMethodPriv$getOutputFilesNames(TaskRun.groovy:295) ~[nextflow-0.11.0.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:368) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:137) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.processor.TaskRun$_closure1.doCall(TaskRun.groovy) ~[nextflow-0.11.0.jar:na]
at nextflow.processor.TaskRun$_closure1.doCall(TaskRun.groovy) ~[nextflow-0.11.0.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:127) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.Closure.call(Closure.java:417) ~[groovy-2.3.7.jar:2.3.7]
at java_util_concurrent_Callable$call.call(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.processor.TaskRun.getOutputFilesNames(TaskRun.groovy) ~[nextflow-0.11.0.jar:na]
at nextflow.processor.TaskRun$getOutputFilesNames$4.call(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.executor.Executor.unstageOutputFilesScript(Executor.groovy:219) ~[nextflow-0.11.0.jar:na]
at nextflow.executor.Executor.unstageOutputFilesScript(Executor.groovy) ~[nextflow-0.11.0.jar:na]
at nextflow.executor.LocalExecutor.createTaskHandler(LocalExecutor.groovy:76) ~[nextflow-0.11.0.jar:na]
at nextflow.executor.LocalExecutor$createTaskHandler.call(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.processor.TaskDispatcher.submit(TaskDispatcher.groovy:145) ~[nextflow-0.11.0.jar:na]
at nextflow.processor.TaskDispatcher$submit$0.call(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.processor.TaskProcessor.submitTask(TaskProcessor.groovy:1313) ~[nextflow-0.11.0.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.processor.TaskProcessor.checkCachedOrLaunchTask(TaskProcessor.groovy:510) ~[nextflow-0.11.0.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153) ~[groovy-2.3.7.jar:2.3.7]
at nextflow.processor.ParallelTaskProcessor.invokeTask(ParallelTaskProcessor.groovy:364) ~[nextflow-0.11.0.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1074) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokePropertyOrMissing(MetaClassImpl.java:1253) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1209) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:807) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:44) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.Script.invokeMethod(Script.java:78) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:414) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:353) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141) ~[groovy-2.3.7.jar:2.3.7]
at Script1$_run_closure1.doCall(Script1.groovy:1) ~[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.7.jar:2.3.7]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.7.jar:2.3.7]
at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.7.jar:2.3.7]
at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120) ~[gpars-1.2.1.jar:1.2.1]
at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35) ~[gpars-1.2.1.jar:1.2.1]
at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58) ~[gpars-1.2.1.jar:1.2.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_51]
Given the example pipeline and configuration below:
// test.nf
process test {
"""
echo Test
"""
}
// nextflow.config
process {
echo = true
$test.container = 'debian'
}
The following error message is displayed:
$ nextflow run test.nf -with-docker
N E X T F L O W ~ version 0.12.5
You request to run with Docker but no image has been specified
I upgraded 0.12.2 to 0.13.1 and now see the error message "Error: cannot find Java or it's a wrong version". 0.12.2 worked and still works fine. 0.13.1 gives this error message.
~ โฏโฏโฏ nextflow -version
Error: cannot find Java or it's a wrong version -- please make sure that Java 7 or higher it's installed
Note: Nextflow is trying to use the Java VM defined by the following environment variables:
JAVA_CMD: /Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home
JAVA_HOME:
~ โฏโฏโฏ java -version โ
java version "1.8.0_40"
Java(TM) SE Runtime Environment (build 1.8.0_40-b25)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
~ โฏโฏโฏ brew switch nextflow 0.12.2
Cleaning /usr/local/Cellar/nextflow/0.12.2
Cleaning /usr/local/Cellar/nextflow/0.13.1
1 links created for /usr/local/Cellar/nextflow/0.12.2
~ โฏโฏโฏ nextflow -version
N E X T F L O W
Version 0.12.2 build 2674
last modified 12-02-2015 11:17 UTC (12:17 CEST)
http://nextflow.io
Dear sir.
I encounter two question, as follow
_1_ . I don`t find the "Continuous checkpoints" function.
processes still Submitted after being sucessfully executed.
=message= [llwang@cu03 example_nextflow]$ nextflow run mytest.nf
=message= N E X T F L O W ~ version 0.12.0
=message= [warm up] executor > local
=message= [1b/f5ade3] Submitted process > validate_utr (1)
=message= [ed/0a1fa0] Submitted process > validate_mir (1)
=message= [3d/9b255f] Submitted process > cor_mir_utr (3)
=message= [fa/3a387a] Submitted process > cor_mir_utr (1)
=message= ...
=message= ...
=message= [88/8e6df7] Submitted process > cor_mir_utr (104)
=message= Result saved to file: /llwang-works/data/ceRNA/example_nextflow/result.txt
...
=message= [llwang@cu03 example_nextflow]$ nextflow run mytest.nf
=message= N E X T F L O W ~ version 0.12.0
=message= [warm up] executor > local
=message= [1b/f5ade3] Submitted process > validate_utr (1)
=message= [ed/0a1fa0] Submitted process > validate_mir (1)
=message= [3d/9b255f] Submitted process > cor_mir_utr (3)
=message= [fa/3a387a] Submitted process > cor_mir_utr (1)
=message= [e6/8b257c] Submitted process > cor_mir_utr (7)
=message= [95/dbd725] Submitted process > cor_mir_utr (5)
_2_ . using pbs executor , the processing stop at the last process...
=message= [llwang@cu03 example_nextflow]$ nextflow run mytest.nf
=message= N E X T F L O W ~ version 0.12.0
=message= [warm up] executor > pbs
=message= [ae/0543aa] Submitted process > validate_utr (1)
=message= [a7/4482ba] Submitted process > validate_mir (1)
=message= [51/abc5ae] Submitted process > cor_mir_utr (4)
=message= [b8/a1b37e] Submitted process > cor_mir_utr (3)
=message= [70/f6827c] Submitted process > cor_mir_utr (9)
=message= [16/af8b6e] Submitted process > cor_mir_utr (103)
=message= [b4/46e6e4] Submitted process > cor_mir_utr (104)
my script as follow
1 // parameters
2 params.mirfile = "/llwang-works/data/ceRNA/example_nextflow/data/expr/mir.expr"
3 params.utrfile = "/llwang-works/data/ceRNA/example_nextflow/data/expr/utr.expr"
4 params.chunkSize = 200
5
6 params.out = "/llwang-works/data/ceRNA/example_nextflow/result.txt"
7
8
9 // Channels
10 mirCh = Channel.fromPath(params.mirfile)
11 utrCh = Channel.fromPath(params.utrfile)
12
13
14 // validate utr sequence
15 process validate_utr {
16 input:
17 file 'source' from utrCh
18
19 output:
20 file 'valid' into utrValCh
21
22 """
23 perl /llwang-works/data/ceRNA/example_nextflow/scripts/validate.pl source > valid
24 """
25 }
26
27
28 // validate microRNA sequence
29 process validate_mir {
30 input:
31 file 'source' from mirCh
32
33 output:
34 file 'valid' into mirValCh
35
36 """
37 perl /llwang-works/data/ceRNA/example_nextflow/scripts/validate.pl source > valid
38 """
39 }
40
41
42 utrValSplitCh = utrValCh.splitText( by: params.chunkSize )
43
44
45 process cor_mir_utr {
46 input:
47 file 'utr' from utrValSplitCh
48 file 'mir' from mirValCh.first()
49
50 output:
51 file 'corel' into corCh
52
53 """
54 Rscript /llwang-works/data/ceRNA/example_nextflow/scripts/cor.R utr mir 0.05 0.7 corel
55 """
56 }
57
58 corCh
59 .collectFile(name: file(params.out))
60 .subscribe {
61 println "Result saved to file: $it"
62 }
63
The code example in the output files section on the process page does not work.
The code xample in question:
process randomNum {
output:
file 'result.txt' into numbers
'''
echo $RANDOM > result
'''
}
numbers.subscribe { println "Received: " + it.text }
When executed, I get:
[14:45:53] ~/code/test/nextflow $ nextflow run randomnum.nf
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=lcd_hrgb -Dswing.aatext=true -Dsun.java2d.xrender=true
N E X T F L O W ~ version 0.9.0
[warm up] executor > local
[47/583d32] Submitted process > randomNum (1)
ERROR: Error executing process > 'randomNum (1)'
Caused by:
Missing output file: 'result.txt' expected by process: randomNum (1)
Command executed:
echo $RANDOM > result
Command exit status:
0
Command output:
(empty)
Work dir:
/home/samuel/code/test/nextflow/work/47/583d329642c9b78d65fadaabf6637a
Tip: view the complete command output by changing to the process work dir and entering the command: 'cat .command.out'
./nextflow http://goo.gl/ngf8dL
N E X T F L O W ~ version 0.8.1
ERROR: java.lang.NoClassDefFoundError: java/util/HashMap$Entry -- See the file '.nextflow.log' for more error details
A dry run option would be great for validating a pipeline script without actually executing any processes.
I have a test.nf
file like the following:
process resume {
output:
val str into out
script:
str = 'test'
command = "echo ${str}"
command += " > test.out"
return command
}
When I run it for the first time everything goes well:
$ nextflow run test.nf
N E X T F L O W ~ version 0.9.0
[warm up] executor > local
[2b/aac531] Submitted process > resume (1)
When I run the pipeline with the -resume
options I get an error:
$ nextflow run test.nf
N E X T F L O W ~ version 0.9.0
[warm up] executor > local
ERROR: Error executing process > 'resume (1)'
Caused by:
Class cannot be created (missing no-arg constructor): groovy.lang.GString$3
Tip: check the log file '.nextflow.log' for more details
Setting return command.toString()
did not help. Here is the stacktrace:
Aug-27 15:01:03.702 [Actor Thread 3] ERROR nextflow.processor.TaskProcessor - Error executing process > 'resume (1)'
Caused by:
Class cannot be created (missing no-arg constructor): groovy.lang.GString$3
Tip: check the log file '.nextflow.log' for more details
Aug-27 15:01:03.706 [Actor Thread 3] DEBUG nextflow.processor.TaskProcessor - Process resume raise the following exception:
com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): groovy.lang.GString$3
at com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1228) ~[kryo-2.24.0.jar:na]
at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1049) ~[kryo-2.24.0.jar:na]
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1058) ~[kryo-2.24.0.jar:na]
at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:547) ~[kryo-2.24.0.jar:na]
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:523) ~[kryo-2.24.0.jar:na]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) ~[kryo-2.24.0.jar:na]
at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143) ~[kryo-2.24.0.jar:na]
at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21) ~[kryo-2.24.0.jar:na]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) ~[kryo-2.24.0.jar:na]
at com.esotericsoftware.kryo.Kryo$readClassAndObject.call(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.6.jar:2.3.6]
at nextflow.util.KryoHelper.deserialize(SerializationHelper.groovy:114) ~[main.jar:na]
at nextflow.util.KryoHelper$deserialize.call(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.6.jar:2.3.6]
at nextflow.processor.DelegateMap.read(DelegateMap.groovy:189) ~[main.jar:na]
at nextflow.processor.DelegateMap$read$0.call(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) ~[groovy-2.3.6.jar:2.3.6]
at nextflow.processor.TaskProcessor.checkCachedOutput(TaskProcessor.groovy:602) ~[main.jar:na]
at nextflow.processor.TaskProcessor$checkCachedOutput$2.callCurrent(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149) ~[groovy-2.3.6.jar:2.3.6]
at nextflow.processor.TaskProcessor.checkCachedOrLaunchTask(TaskProcessor.groovy:482) ~[main.jar:na]
at nextflow.processor.TaskProcessor.checkCachedOrLaunchTask(TaskProcessor.groovy) ~[main.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_25]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_25]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_25]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_25]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153) ~[groovy-2.3.6.jar:2.3.6]
at nextflow.processor.ParallelTaskProcessor.invokeTask(ParallelTaskProcessor.groovy:338) ~[main.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_25]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_25]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_25]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_25]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1074) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.MetaClassImpl.invokePropertyOrMissing(MetaClassImpl.java:1253) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1209) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:807) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:44) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.Script.invokeMethod(Script.java:78) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:414) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:353) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141) ~[groovy-2.3.6.jar:2.3.6]
at Script1$_run_closure1.doCall(Script1.groovy:1) ~[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_25]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_25]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_25]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_25]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324) ~[groovy-2.3.6.jar:2.3.6]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:278) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016) ~[groovy-2.3.6.jar:2.3.6]
at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.6.jar:2.3.6]
at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120) ~[gpars-1.2.1.jar:1.2.1]
at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35) ~[gpars-1.2.1.jar:1.2.1]
at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58) ~[gpars-1.2.1.jar:1.2.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
Aug-27 15:01:03.707 [Actor Thread 3] DEBUG nextflow.Session - Session abort -- terminating all processors
Aug-27 15:01:03.708 [Actor Thread 1] DEBUG n.processor.ParallelTaskProcessor - <resume> After stop -- shareObjs [:]
Aug-27 15:01:03.709 [main] DEBUG nextflow.Session - <<< phaser deregister (session)
Any of the following actions solves the issue:
output
declaration blockcommand = "echo ${str} > test.out"
...
command = "echo ${str}".toString()
command += " > test.out".toString()
return command
...
+=
:...
command = "echo ${str}"
command = "${command} > test.out"
return command
...
Calling nextflow run hello
gets stuck with the message Pulling nextflow-io/hello ...
.
A common problem with my work environment is that I need to configure a proxy. Apparently, nextflow does not pick up the environment variables HTTP_PROXY
and HTTPS_PROXY
. How can I configure a proxy to use for the nextflow run hello
command?
I have file a.txt, it has 20 lines.
then I want use variable with value 20 in nextflow process.
I tried using nextflow script below, but failed.
How should I do?
1 // initialization
2 x=5
3
4 // setting again
5 x=10
6
7 // setting again in process
8 process set{
9
10 script:
11 x = " wc -l a.txt | awk '{print $1}' "
12 }
13
14
15 process get{
16 echo true
17
18
19 script:
20 """
21 echo "variable x in use : "${x}
22 """
23 }
For example
#!/bin/env nextflow
params.in = 'test.txt'
test_file = file(params.in)
name = test_file.getBaseName()
process test {
input:
file test_file
val x from name
output:
file "${x}.tsv" into out
"""
echo "${test_file}" > ${x}.tsv
"""
}
out.subscribe {
println "File name: $it.name"
println "File content: $it.text"
}
When resumed throws the following exception
Jun-05 10:55:59.004 [main] DEBUG nextflow.script.CliRunner - nextflow test.nf -resume
Jun-05 10:55:59.108 [main] DEBUG nextflow.script.CliRunner - Parsing config file: /Users/pditommaso/.nextflow/config
Jun-05 10:55:59.455 [main] DEBUG nextflow.Session - Executor pool size: 9
Jun-05 10:55:59.477 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /Users/pditommaso/workspace/nextflow/bin
Jun-05 10:55:59.478 [main] DEBUG nextflow.script.CliRunner - Script bin dir: null
Jun-05 10:55:59.515 [main] DEBUG nextflow.script.CliRunner -
Version: 0.8.4-b1 build 1838
Last modified: 04-06-2014 12:59 UTC (14:59 CEST)
Os: Mac OS X 10.9.3
Groovy: 2.3.0
Jvm: Oracle Corporation 1.7.0_51-b13
Opts: -Djava.awt.headless=true -Xms512m
Encoding: UTF-8 (UTF-8)
Address: Retina.local [172.17.4.167]
File systems: file,jar
Jun-05 10:55:59.526 [main] DEBUG nextflow.Session - > Session start -- phaser register (session)
Jun-05 10:55:59.534 [main] DEBUG nextflow.processor.TaskDispatcher - Dispatcher > start
Jun-05 10:55:59.534 [main] DEBUG nextflow.script.CliRunner - > Script parsing
Jun-05 10:55:59.598 [main] DEBUG nextflow.Session - Using default localLib path: /Users/pditommaso/workspace/nextflow/lib
Jun-05 10:55:59.605 [main] DEBUG nextflow.script.CliRunner - Adding to the classpath library: /Users/pditommaso/workspace/nextflow/lib
Jun-05 10:55:59.606 [main] DEBUG nextflow.script.CliRunner - Adding to the classpath library: /Users/pditommaso/workspace/nextflow/lib/commons-cli-1.2.jar
Jun-05 10:55:59.607 [main] DEBUG nextflow.script.CliRunner - Adding to the classpath library: /Users/pditommaso/workspace/nextflow/lib/guava-cache-16.0.1.jar
Jun-05 10:55:59.767 [main] DEBUG nextflow.script.CliRunner - > Launching execution
Jun-05 10:55:59.887 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: null
Jun-05 10:55:59.887 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-05 10:55:59.904 [main] INFO nextflow.executor.Executor - [warm up] executor > local
Jun-05 10:55:59.940 [main] DEBUG n.processor.TaskPollingMonitor - Creating task monitor for executor 'local' > capacity: 7; pollInterval: 100ms; dumpInterval: 5min
Jun-05 10:55:59.952 [main] DEBUG nextflow.processor.TaskDispatcher - Starting monitor: TaskPollingMonitor
Jun-05 10:55:59.953 [main] DEBUG n.processor.TaskPollingMonitor - >>> phaser register (scheduler)
Jun-05 10:56:00.032 [main] DEBUG nextflow.script.BaseParam - output > channel unknown: out -- creating a new instance
Jun-05 10:56:00.065 [main] DEBUG n.processor.ParallelTaskProcessor - Creating operator > test -- maxForks: 9
Jun-05 10:56:00.082 [main] DEBUG nextflow.Session - >>> phaser register (process)
Jun-05 10:56:00.102 [main] DEBUG nextflow.script.CliRunner - > Await termination
Jun-05 10:56:00.200 [Actor Thread 3] ERROR nextflow.processor.TaskProcessor - Error executing process > 'test (1)'
Caused by:
x
Tip: check the log file '.nextflow.log' for more details
Jun-05 10:56:00.204 [Actor Thread 3] DEBUG nextflow.processor.TaskProcessor - Process test raise the following exception:
groovy.lang.MissingPropertyException: x
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_51]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190) ~[groovy-2.3.0.jar:2.3.0]
at nextflow.script.BaseParam.resolveName(ParamsIn.groovy:200) ~[main.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149) ~[groovy-2.3.0.jar:2.3.0]
at nextflow.script.FileOutParam$_getFilePatterns_closure1.doCall(ParamsOut.groovy:233) ~[main.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.0.jar:2.3.0]
at nextflow.script.TokenGString$_resolve_closure1.doCall(ScriptTokens.groovy:130) ~[main.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.Closure.call(Closure.java:439) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:2342) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:2312) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.dgm$63.invoke(Unknown Source) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.0.jar:2.3.0]
at nextflow.script.TokenGString.resolve(ScriptTokens.groovy:130) ~[main.jar:na]
at nextflow.script.TokenGString$resolve.call(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.0.jar:2.3.0]
at nextflow.script.FileOutParam.getFilePatterns(ParamsOut.groovy:233) ~[main.jar:na]
at nextflow.script.FileOutParam$getFilePatterns$0.call(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.0.jar:2.3.0]
at nextflow.processor.TaskProcessor.collectOutFiles(TaskProcessor.groovy:1008) ~[main.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153) ~[groovy-2.3.0.jar:2.3.0]
at nextflow.processor.TaskProcessor$_collectOutputs_closure18.doCall(TaskProcessor.groovy:944) ~[main.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.Closure.call(Closure.java:439) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1373) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1366) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.dgm$149.invoke(Unknown Source) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-2.3.0.jar:2.3.0]
at nextflow.processor.TaskProcessor.collectOutputs(TaskProcessor.groovy:936) ~[main.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153) ~[groovy-2.3.0.jar:2.3.0]
at nextflow.processor.TaskProcessor.checkCachedOutput(TaskProcessor.groovy:617) ~[main.jar:na]
at nextflow.processor.TaskProcessor$checkCachedOutput$2.callCurrent(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149) ~[groovy-2.3.0.jar:2.3.0]
at nextflow.processor.TaskProcessor.checkCachedOrLaunchTask(TaskProcessor.groovy:486) ~[main.jar:na]
at nextflow.processor.TaskProcessor.checkCachedOrLaunchTask(TaskProcessor.groovy) ~[main.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:153) ~[groovy-2.3.0.jar:2.3.0]
at nextflow.processor.ParallelTaskProcessor.invokeTask(ParallelTaskProcessor.groovy:336) ~[main.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1085) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:952) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokePropertyOrMissing(MetaClassImpl.java:1131) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1087) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:730) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:44) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.Script.invokeMethod(Script.java:78) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:407) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:346) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141) ~[groovy-2.3.0.jar:2.3.0]
at Script1$_run_closure1.doCall(Script1.groovy:1) ~[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) ~[groovy-2.3.0.jar:2.3.0]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) ~[groovy-2.3.0.jar:2.3.0]
at groovy.lang.Closure.call(Closure.java:423) ~[groovy-2.3.0.jar:2.3.0]
at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120) ~[gpars-1.2.1.jar:1.2.1]
at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35) ~[gpars-1.2.1.jar:1.2.1]
at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58) ~[gpars-1.2.1.jar:1.2.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_51]
Jun-05 10:56:00.205 [Actor Thread 3] DEBUG nextflow.Session - Session abort -- terminating all processors
Jun-05 10:56:00.206 [Actor Thread 1] DEBUG n.processor.ParallelTaskProcessor - <test> After stop -- shareObjs [:]
Jun-05 10:56:00.207 [main] DEBUG nextflow.Session - <<< phaser deregister (session)
I haven't experienced this myself, so this might be invalid, but SLURM can suspend jobs on a cluster if a high-priority job comes in. This might become a real situation for us in the future, which is why I am concerned about this. The job then goes into the suspended state, indicates by the S
flag, as indicated here https://computing.llnl.gov/linux/slurm/squeue.html.
I looked into the code, and there's no handling of the S
flag.
I cannot test a fix myself, unfortunately, but to me it seems to be safe to treat the S flag same as running, meaning Nextflow should keep waiting for it to finish?
Looks like I won't have to wait long for the dir globbing -- I'm having problems getting it working though :
Here is my Channel declaration ๐
samps = Channel.path('/data/bifx/Sequence_data/TXOME/Project*/*', type:'dir')
and here's what happens when I try to run it:
[root@szdedicated /data/bifx/tlaurent/trinity_wd]# ./nextflow run_trinity.nf
N E X T F L O W ~ version 0.7.3
ERROR: groovy.lang.MissingMethodException: No signature of method: static nextflow.Channel.path() is applicable for argument types: (java.util.LinkedHashMap, java.lang.String) values: [[type:dir], /data/bifx/Sequence_data/TXOME/Project*/*]
Possible solutions: path(java.lang.String), path(java.lang.Object), path(java.util.regex.Pattern), watch(java.lang.String), watch(java.lang.String, java.lang.String), watch(java.util.regex.Pattern, java.lang.String) -- See the file '.nextflow.log' for more error details
BTW I did pull the new package again:
Thanks,
For example:
process Hola {
input:
each x from 1,2
output:
set 'file_x', 'file_y' into results
'''
touch file_x
touch file_y
'''
}
While it should be possible to perform a left outer join operation using map
and reduce
operators, it might be useful to include a leftJoin
operator in the DataflowExtensions groovy class.
Here are some simple tests
def testLeftJoin() {
when:
def ch1 = Channel.from( 1,2,3 )
def ch2 = Channel.from( 1,0,0,2,7,8,9,3 )
// forward
def result = ch1.leftJoin(ch2)
then:
result.val == [1,1]
result.val == [2,2]
result.val == [3,3]
result.val == Channel.STOP
}
def testLeftJoin2() {
when:
def ch1 = Channel.from( 1,2,3 )
def ch2 = Channel.from( 1,0,0,2,7,8,9,3 )
// reverse
def result = ch2.leftJoin(ch1)
then:
result.val == [1,1]
result.val == [0,null]
result.val == [0,null]
result.val == [2,2]
result.val == [7,null]
result.val == [8,null]
result.val == [9,null]
result.val == [3,3]
result.val == Channel.STOP
// alternatively, it would be ok if order was not preserved
/*
result.val == [1,1]
result.val == [2,2]
result.val == [3,3]
result.val == [0,null]
result.val == [0,null]
result.val == [7,null]
result.val == [8,null]
result.val == [9,null]
result.val == Channel.STOP
*/
}
Using process.scratch=true
locally on a Mac crashes the pipeline run
Currently pipelines running on cirrus (or is it gridgain?) have the option of using S3 for the nextflow workdir. This kind of works outside of cirrus by setting 'nextflow run -w s3://bucket', with nextflow indeed using the designated S3 object as the work directory, except that for some reason pipelines are unable to finish(?). Why this is the case was discussed briefly in the gitter chat but I forget the cause.
Anyway, this would be a really nice feature to have to complement using S3 for inputs/outputs directly within Nextflow. An example use-case would be running nextflow on an EC2 cluster (via StarCluster), and running a nextflow pipeline such that all inputs, outputs and contents of the workdir were stored in S3.
I just had a situation where I removed some code, removing one of the input channels for a process.
What happened next is that my nextflow script would just sit there at the stage where it would normally launch said process, not giving any warning (that I can find) and not doing anything at all.
I have experienced similar issues before, so I know I had to check my channels, but for a person new to Nextflow such a thing can be quite confusing, or even a dealbreaker.
What I'm talking about is
process someProcess {
input:
file bam_file from bam_input
file vcf_file from vcf_input
file idx_file from idx_input <-- channel doesn't exist anymore
etc...
Can you confirm this behaviour? Is it possible to check consistency upon initialization and raise an exception in those cases?
I'm running an nf file with multiple processes that use storeDir
and file inputs. Exception is non-deterministic and only occurs with over 5+ proc declarations.
race.nf:
#!/usr/bin/env nextflow
seqs = file('seqs/*.fastq')
process proc1 {
storeDir 'results'
input:
file fastq from seqs
output:
set val(baseName), file(outName) into proc1
script:
baseName = fastq.baseName
outName = baseName + '.proc1'
"""
echo "blarg" > $outName
"""
}
process proc2 {
storeDir 'results'
input:
file fastq from seqs
output:
set val(baseName), file(outName) into proc2
script:
baseName = fastq.baseName
outName = baseName + '.proc2'
"""
echo "blarg" > $outName
"""
}
... repeat as necessary
output:
N E X T F L O W ~ version 0.13.3
Launching race.nf
[warm up] executor > local
[skipping] Stored process > proc2 (3)
[skipping] Stored process > proc3 (1)
[skipping] Stored process > proc3 (3)
[skipping] Stored process > proc2 (2)
[skipping] Stored process > proc2 (1)
[skipping] Stored process > proc1 (2)
[skipping] Stored process > proc5 (1)
[skipping] Stored process > proc5 (2)
[skipping] Stored process > proc5 (3)
[skipping] Stored process > proc1 (1)
[skipping] Stored process > proc1 (3)
[skipping] Stored process > proc3 (2)
[skipping] Stored process > proc4 (1)
[skipping] Stored process > proc4 (2)
[skipping] Stored process > proc4 (3)
ERROR ~ Error executing process > 'proc5 (1)'
Caused by:
Cannot invoke method bindOutput() on null object
-- Check '.nextflow.log' file for details
Uncaught exception occurred in gpars pool Actor Thread 14
java.util.concurrent.RejectedExecutionException: Task groovyx.gpars.actor.AbstractLoopingActor$1@2766ae7b rejected from java.util.concurrent.ThreadPoolExecutor@ad4cdb3[Shutting down, pool size =
1, active threads = 1, queued tasks = 0, completed tasks = 49]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at groovyx.gpars.scheduler.DefaultPool.execute(DefaultPool.java:155)
at groovyx.gpars.util.AsyncMessagingCore.store(AsyncMessagingCore.java:99)
at groovyx.gpars.actor.AbstractLoopingActor.send(AbstractLoopingActor.java:241)
at groovyx.gpars.dataflow.operator.DataflowProcessorActor.send(DataflowProcessorActor.java:85)
at groovyx.gpars.actor.AbstractLoopingActor.send(AbstractLoopingActor.java:241) [0/1821]
at groovyx.gpars.dataflow.operator.DataflowProcessorActor.send(DataflowProcessorActor.java:85)
at groovyx.gpars.actor.AbstractLoopingActor.terminate(AbstractLoopingActor.java:215)
at groovyx.gpars.dataflow.operator.DataflowProcessor.terminate(DataflowProcessor.java:147)
at nextflow.Session.abort(Session.groovy:375)
at nextflow.Session$abort$7.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at nextflow.processor.TaskProcessor.resumeOrDie(TaskProcessor.groovy:756)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
at nextflow.processor.TaskProcessor.handleException(TaskProcessor.groovy:694)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1206)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrentN(ScriptBytecodeAdapter.java:80)
at nextflow.processor.ParallelTaskProcessor.this$dist$invoke$2(ParallelTaskProcessor.groovy)
at nextflow.processor.ParallelTaskProcessor$TaskProcessorInterceptor.methodMissing(ParallelTaskProcessor.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaClassImpl.invokeMissingMethod(MetaClassImpl.java:932)
at groovy.lang.MetaClassImpl.invokePropertyOrMissing(MetaClassImpl.java:1255)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1208)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1015)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
at nextflow.processor.ParallelTaskProcessor$TaskProcessorInterceptor.onException(ParallelTaskProcessor.groovy:296)
at groovyx.gpars.dataflow.operator.DataflowProcessor.fireOnException(DataflowProcessor.java:331)
at groovyx.gpars.dataflow.operator.DataflowProcessor.reportError(DataflowProcessor.java:293)
at groovyx.gpars.dataflow.operator.DataflowProcessorActor.reportException(DataflowProcessorActor.java:125)
at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:122)
at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.access$001(ForkingDataflowOperatorActor.java:35)
at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor$1.run(ForkingDataflowOperatorActor.java:58)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
It would be great to be able to log into a docker registry ("login" CLI command). This permits accessing private registries that authorization, as well as private repos on Docker Hub.
When a configuration file includes another config which declares config attributes using a dot property syntax an exception is raised.
For example having nextflow.config
like this:
includeConfig 'foo.config'
and foo.config
like the following:
params.bar = 1
The following error message is displayed:
ERROR ~ Not such variable: params
Dear Sir,
I have two parallel process, utr corelatted by the miRNA and lncRNA corelatted by the miRNA.
But only corelation of utr and miRNA executted.
== please see the zip attachment "example_nextflow.zip.png" ==
[llwang@fat01 example_nextflow]$ ./corelation.nf
== messages == N E X T F L O W ~ version 0.12.0
== messages == [warm up] executor > local
== messages == [3a/f8e09d] Submitted process > val_mir (1)
== messages == [98/77d5d4] Submitted process > val_lnc (1)
== messages == [ce/1d8363] Submitted process > val_utr (1)
== messages == [55/5bc0ca] Submitted process > cor_utr (5)
== messages == [df/44885b] Submitted process > cor_utr (6)
== messages == [17/0ce5df] Submitted process > cor_utr (2)
== messages == [a3/2d4d29] Submitted process > cor_utr (1)
== messages == [ae/480a17] Submitted process > cor_utr (4)
...
== messages == [ab/976891] Submitted process > cor_utr (99)
== messages == [db/41a6be] Submitted process > cor_utr (95)
== messages == [b0/9ff97f] Submitted process > cor_utr (100)
== messages == Result saved to file: /llwang-works/installation/example_nextflow/utr.cor.result
*************** ***************** ******************
* utr channel * * miRNA channel * * lncRNA channel *
*************** ***************** ******************
| | |
validate validate validate
| | |
\./ \./ \./
. . .
***************** ***************** ******************
* utr val chann * * mir val chann * * lnc val chann *
***************** ***************** ******************
| | | |
split | | split
| | | |
\./ \./ \./ \./
. . . .
**************************** ****************************
* corelation channel * * corelation channel *
**************************** ****************************
| |
collect collect
| |
\./ \./
. .
file_utr file_lnc
Add a method all()
to params
object that renders a string containing all command line parameters.
For example, having the following params
map:
[a: 1, b: 2, c: 3]
It will return a string:
--a 1 --b 2 --c 3
Add eventually some option to control the option prefix (-
or --
), the name-value separator (blank, or =
) and to includes, excludes entries, quote values
Implement a new command that allows the user to selectively clean the work/scratch directory.
Ideally the user should be able to select which runs have to discarded and keep the others. Currently to each pipeline execution is associated a UUID number. The history of all runs is stored in the file .nextflow.history
(in the working directory). It can be viewed with the command nextflow history
. It prints a report similar to the following (the oldest run is showed at the top, the latest at the bottom):
a77d4b73-8fdc-4111-bb19-31e30a7248d3 nextflow run piper
aa4fd46b-fee7-40b4-bff6-e3acd5950254 nextflow run piper --genome-file /path/to/file
49685297-1e52-41d5-9e0c-f00d09d1d17a nextflow run grape --read1 '*_1.fa' --read2 '*_2.fa'
cbf5e933-f725-4a1f-a714-a0c8715bff74 nextflow run grape --annot /path/to/annotation.gff
It could be possibile to implement a new command clean
command that delete all the files produced by a pipeline execution having a specified UUID number, or before that execution, or after that execution.
For example:
nextflow clean -before <UUID>
Delete all files produced before the pipeline run with the specified UUID
nextflow clean -after <UUID>
Delete all files produced after the pipeline run with the specified UUID
nextflow clean -run <UUID>
Delete all files produced by the pipeline run with the specified UUID
nextflow clean -but <UUID>
Delete all files expect the ones produced by the runs specified
nextflow clean
By default delete all files expect the ones produced by the last run.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.