askimed / nf-test

Simple test framework for Nextflow pipelines

🚀 nf-test

Simple test framework for Nextflow pipelines The full documentation can be found here.

Installation

nf-test has the same requirements as Nextflow and can be used on POSIX compatible systems like Linux or OS X. You can install nf-test using the following command:

curl -fsSL https://code.askimed.com/install/nf-test | bash

If you don't have curl installed, you could use wget:

wget -qO- https://code.askimed.com/install/nf-test | bash

It will create the nf-test executable file in the current directory. Optionally, move the nf-test file to a directory accessible by your $PATH variable.

Conda

To install this package from Bioconda run the following command:

conda install -c bioconda nf-test

Note: this recipe is not maintained by us.

Compile from source

To compile nf-test from source you shall have maven installed. This will produce a nf-test/target/nf-test.jar file.

git clone [email protected]:askimed/nf-test.git
cd nf-test
mvn install

To use the newly compiled nf-test.jar, update the nf-test bash script that is on your PATH to point to the new .jar file. First locate it with which nf-test, and then modify APP_HOME and APP_JAR vars at the top:

#!/bin/bash
APP_HOME="/PATH/TO/nf-test/target/"
APP_JAR="nf-test.jar"
APP_UPDATE_URL="https://code.askimed.com/install/nf-test"
...

Usage

nf-test test example/*.nf.test

The full documentation can be found here.

Badge

Show the world your Nextflow pipeline is using nf-test:

[![nf-test](https://img.shields.io/badge/tested_with-nf--test-337ab7.svg)](https://github.com/askimed/nf-test)

Contact

• Lukas Forer @lukfor
• Sebastian Schönherr @seppinho

nf-test's Issues

Nevermind: Using libraries outside of then block

Hi! I have a file named Util.groovy which contains functionality I would like to share between tests. According to the documentation, placing this file in the lib directory allows it to be accessed from within the then block — very nice!

However, I would like to use some of the util methods to prepare the environment before running the tested process (e.g. creating a temporary directory to write the results to). I have tried calling them from various scopes: inside the when block, within the test() block before the when, at the very top of the file; Util doesn't seem to be available to code at any other point in the .nf.test file.

Explicitly importing the class doesn't resolve either. Perhaps this is a problem with the classpath but I can't find a way to change the classpath.

Is there any way to achieve what I'm trying to do here?

Update: immediately after posting this I see that the documentation lists this feature as experimental... My bad.

Stub run

Is it possible to run Nextflow with -stub-run mode within nf-test? In some cases where we don't really care about the pipeline outputs (eg. see #60) it would speed up the tests a lot.

Thanks!

Extending PATH with bin/ when testing processes

Hi @lukfor, thanks again for the great work on this. I encountered what I believe is a bug in the way Nextflow is invoked when running process-level tests. In my case, I have scripts in bin/ that Nextflow automatically adds to the PATH at runtime. When testing a process that calls one of the scripts in bin/, it fails because it can't find the script on the PATH but only when testing an individual process. When testing workflows and pipeline runs it works fine. It appears that the mock.nf generated for the test is invoked from a different directory than the one specified by $baseDir (the value of which is as expected). As such, it is not possible for me to test processes that invoke custom scripts.

I'm currently running [email protected] in case this has been fixed in one of the recent releases.

Testing config resolution at task level

Hi there,

Thanks for nf-test, having a lot of fun with it in @nf-core and it's working well. Apologies if this is a naive question, still getting to grips with the concepts.

In the pipeline I'm working on at the moment I have a particularly horrendous chunk of Nextflow config and I'd like to simply test that the config resolution works in the way that I expect. In other words, when I provide these params, I want to check that process.withName: TRIMGALORE { ext.args } contains this substring. It would also be fine to check that the generated .command.sh contains a certain substring.

Is there an existing way to dig into the resolved task details like this?

Phil

outputDir point to the full path

outputDir point to the full path
I'm trying to have params.outdir to be relative, and not really any idea on how to do so.
Any solutions to get outputDir to be relative?
Or any other variable that can be used for that?

This is what I have:

cat /home/maxime/workspace/nf-core_sarek/.nf-test/tests/6b4fc92d7235947cf7ae249c9cd9ebf/meta/params.json

{"baseDir":"/home/maxime/workspace/nf-core_sarek","outputDir":"/home/maxime/workspace/nf-core_sarek/.nf-test/tests/6b4fc92d7235947cf7ae249c9cd9ebf/output","outdir":"/home/maxime/workspace/nf-core_sarek/.nf-test/tests/6b4fc92d7235947cf7ae249c9cd9ebf/output"}

This is what I'm trying to do:

cat /home/maxime/workspace/nf-core_sarek/.nf-test/tests/6b4fc92d7235947cf7ae249c9cd9ebf/meta/params.json

{"baseDir":"/home/maxime/workspace/nf-core_sarek","outputDir":".nf-test/tests/6b4fc92d7235947cf7ae249c9cd9ebf/output","outdir":".nf-test/tests/6b4fc92d7235947cf7ae249c9cd9ebf/output"}

I have some files that are created with using params.outdir and it messes up the checksum with the full path.

new nested params `load` feature breaks workflow

@lukfor The support for nested params and loading them from external files (#50, fixing #15) is a very welcome new feature, thank you!

However, I am confused by something -- it seems like the load() method is injecting baseDir and outputDir into my params. And not just at the top level but in nested maps too. Example:

Here is my main.nf.test file:

nextflow_pipeline {
    name "Pipeline Quick Integration Test main.nf"
    script "main.nf"
    test("Should run without failures") {
        when {
            params {
                load("$baseDir/tests/params/params_debug.json")
            }
        }
        then {
            println(params)
            assert workflow.success
        }
    }
}

Here is my params_debug.json file:

{
  "references": {
    "GRCh38": {
      "description": "Homo sapiens (GRCh38.p13)",
      "bwa_index": "path/to/GRCh38.fa.bwt",
    }
  },
  "output_dir": "tests/output/smoke/",
  "samples": [
    {
      "id": "SAMP1",
      "name": "Sample 1",
      "filepath_r1": "tests/data/reads/SAMP1_R1_001_10k.fastq",
      "filepath_r2": "tests/data/reads/SAMP1_R3_001_10k.fastq",
      "filepath_umi": "tests/data/reads/SAMP1_R2_001_10k.fastq",
      "reference": "GRCh38"
    }
  ]
}

Here is the (pretty printed) output of that println call in my main.nf.test file:

[
  baseDir:/work/mypipeline,
  outputDir:/work/mypipeline/.nf-test/tests/42062baab057bce9c1028ff0ac9ef88b/output,
  references: [
    baseDir:/work/mypipeline,
    outputDir:/work/mypipeline/.nf-test/tests/42062baab057bce9c1028ff0ac9ef88b/output,
    GRCh38: [
      baseDir:/work/mypipeline,
      outputDir:/work/mypipeline/.nf-test/tests/42062baab057bce9c1028ff0ac9ef88b/output,
      description:Homo sapiens (GRCh38.p13),
      bwa_index:path/to/GRCh38.fa.bwt
    ]
  ],
  output_dir:tests/output/smoke/,
  samples:[
    [
      id:SAMP1,
      name:Sample 1,
      filepath_r1:tests/data/reads/SAMP1_R1_001_10k.fastq,
      filepath_r2:tests/data/reads/SAMP1_R3_001_10k.fastq,
      filepath_umi:tests/data/reads/SAMP1_R2_001_10k.fastq, reference:GRCh38
    ]
  ]
]

The thing is, my workflow expects params.references to be a map with a particular structure. So the injection of baseDir and outputDir makes my workflow choke.

It seems like this injection into nested maps is by design, but I am left scratching my head. Is it really necessary to do this to all nested maps? I imagine this could break any workflow that makes assumptions about nested maps structure.

(cc: @birnbera)

There is a problem with the function getMd5

getMd5 function omits the leading 0. For those MD5 starts with 0, it will get the error result. Modify maybe something like this: return bigInt.toString(16).padLeft(32, '0');

JUnit Output

Would be really great if it was possible to optionally output test results in xunit format - as this would make interfacing with reporting in CI systems, (eg bitbucket) easier.

For example, it could run: nf-test test --xunit test_results.xml example/*.nf.test

Working example of 'workflow' tests

Posting the issue discussed here:

Please can you share a working example of a test case for workflow objects? No matter how we try following the documentation, we get a groovy.lang.MissingMethodException:

trial.nf:

#!/usr/bin/env nextflow
nextflow.enable.dsl=2

process sayHello {
    input:
        val cheers

    output:
        stdout emit: verbiage

    script:
    """
    echo -n $cheers
    """
}

workflow trial {
    take: things
    main:
        sayHello(things)
        sayHello.out.verbiage.view()
}

workflow {
    Channel.from(['a']) | trial
}

trial.nf.test:

nextflow_workflow {

    name "Test workflow"
    script "trial.nf"
    workflow "trial"

    test("Should run without failures") {
        when {
            params {
                outdir = "tests/results"
            }
            workflow {
                """
                input[0] = ["a"]
                """
            }
        }

        then {
            //check if test case succeeded
            assert workflow.success
        }
    }
}

Output:

$ ./nf-test test trial.nf.test

🚀 nf-test 0.5.1
https://code.askimed.com/nf-test
(c) 2021 - 2022 Lukas Forer and Sebastian Schoenherr


Test workflow

  Test [57acc7c1] 'Should run without failures' FAILED (0.01s)

  groovy.lang.MissingMethodException: No signature of method: trial_nf$_run_closure1$_closure2.workflow() is applicable for argument types: (trial_nf$_run_closure1$_closure2$_closure3$_closure6) values: [trial_nf$_run_closure1$_closure2$_closure3$_closure6@4721d212]



FAILURE: Executed 1 tests in 0.018s (1 failed)

Feature request: nextflow options

Hi,

Would it be possible to add an option to modify the nextflow command?
I'd like to be able to add stuff like dump-channels to the command line to make debugging easier

Thanks
M

Feature request: support Map objects in workflow outputs.

We use Maps in our workflow channels. When these are parsed from the output_outputCh.json file by the nf-test process by the JsonSlurper, they are read as LazyMap objects. However, these are not recognised by the compareObjects method, resulting an error message littering the output:

  Test [f087ff85] 'Test with all positive input' Warning: Unsupported classes: class org.apache.groovy.json.internal.LazyMap vs. class org.apache.groovy.json.internal.LazyMap
PASSED (4.962s)

It would be desirable if Map objects could be supported in channels. If you are happy for this feature to be added, I'm happy to implement it.

Unit Test Nextflow Functions

It would be great to be able to unit test nextflow functions in a similar way to processes/workflows. In addition to return values, the stdout/stderr could be captured and compared. For example:

functions.nf

def say_hello(name) {
    if (name == null) {
        error('Cannot greet a null person')
    }
        
    def greeting = "Hello ${name}"

    println(greeting)
    return greeting
}

functions.nf.test

nextflow_function {

  name "Test Function Say Hello"

  script "example/functions.nf"
  function "say_hello"

  test("Passing case") {

    when {
      function {
        """
        input[0] = "aaron"
        """
      }
    }

    then {
      assert function.success
      assert function.return == "Hello aaron"
      assert function.stdout == "Hello aaron"
      assert function.stderr == ""
    }

  }

  test("Failure Case") {

    when {
      function {
        """
        input[0] = null
        """
      }
    }

    then {
      assert process.failed
      assert function.stderr.contains("Cannot greet a null person")
    }
  }
}

Nested params

Thank you for creating this package! I love the simplicity and flexibility. Unfortunately, I have run into an issue that will make it difficult to use nf-test in practice. Most Nextflow pipelines have some degree of nesting in their params, e.g.:

{
    "outer": {
        "inner": "value"
    }
}

When running a test that includes nested params, I get an error. As an example:

nextflow_process {

    name "Test Process copy"
    script "main.nf"
    process "copy"

    test("Should run without failures") {

        when {
            params {
                output_dir = "tests/results"
                outer {
                    inner = 'value'
                }
            }
            process {
                """
                input[0] = file("test-file.txt")
                """
            }
        }

        then {
            assert process.success
            with(process.out.newfile) {
                assert size() == 1
                with(file(get(0))) {
                    assert name == 'someotherfilename.txt'
                    assert text == file('test-file.txt').text + 'extra\n'
                }
            }
        }
    }
}

Running this, I get:

groovy.lang.MissingMethodException: No signature of method: main_copy_nf$_run_closure1$_closure2.outer() is applicable for argument types: (main_copy_nf$_run_closure1$_closure2$_closure3$_closure5$_closure7) values: [main_copy_nf$_run_closure1$_closure2$_closure3$_closure5$_closure7@6093d508]
  Possible solutions: use([Ljava.lang.Object;), use([Ljava.lang.Object;), grep(), every(), dump(), grep()

Am I specifying the params incorrectly or are they not yet fully supported within nf-test? Also, is it possible to pass params as a file (i.e. JSON or YAML) rather than as code?

nf-test hang on very simple workflow tests

nf-test can hang when running a test on a very simple workflow. Here's the code to replicate the issue:

$ cat example/meaningless_workflow.nf
workflow PipeWf {
    take:
        inputCh

    main:
        inputCh
            | set { outputCh }

    emit:
        outputCh
}

$ cat example/meaningless_workflow.nf.test 

nextflow_workflow {

    name "Test workflow"
    script "example/meaningless_workflow.nf"
    workflow "PipeWf"

    test("PipeWf will hang") {

        when {
            workflow {
                """
                input[0] = Channel.from([
                  [
                    ["patientID": "patientA"],
                    'test_file_1.txt'
                  ]
                ])
                """
            }
        }

        then {
              assert workflow.success
            }

    }

}

$ nf-test test example/meaningless_workflow.nf.test
# This will hang

Command 'ps' required by nextflow to collect task metrics cannot be found

Hi,

nf-test looks like a very nice tool. I wanted to test a nextflow pipeline with singularity containers.
Since most containers do not have "ps" installed, the tests don't pass.
Is there a way to run the tests without "ps" or are you aware of any other way to make this work?

Thank you!

Bug report: `nf-test` stops at first failure

Hi,

The output of nf-test only show the first assertion failure even if there are multiple assertion failures. It would be really helpful and a time saver if the output would show all assertion failures at once. I hope this won't be too hard to implement.

Cheers,
Nicolas

Bug in importing custom class

I tried example of Third-Party library file given on https://code.askimed.com/nf-test/assertions/libraries/
Getting below error:

groovy.lang.MissingMethodException: No signature of method: static MyWordUtils.capitalize() is applicable for argument types: (String) values: [world]
  Possible solutions: capitalize(java.lang.String)

Checking output files of a pipeline

Hi! First of all thank you for creating this awesome tool.
I do have one question, though: is it possible to check the output files from a pipeline, similar to how you can check the output of modules and workflows?

e.g. I use the $params.outputDir parameter automatically created by nf-test to define the output directory. Is it possible that the tests check the files in this directory along with their md5 checksums?

Is it possible to have dummy output

I want to test a workflow but it has multiple steps. I don't want to test everything in one shot. Is is possible to create some output files and nf-test just run portion for which files are not available (similar to nextflow)

Testing individual processes within a pipeline

We know a pipeline can be tested, and so can individual processes in separate module files. But can I get an individual test for each process within a pipeline? We have a large nextflow script we were trying to test. The processes are not in individual files, but we want to have an individual test for each.

Feature Request: Add tag system

@nvnieuwk:

Could it also be possible to add a tag system (similar to pytest --tag). This would make it easier to run the needed tests on Github Actions instead of having to supply the whole file path.

See #29

Suggestions from `nf-core`

Hi again,

After a discussion in the nf-core community we came up with a couple more suggestions that could make incorporation of nf-test into nf-core/modules a bit easier to implement.

1. Could it be possible to supply a .yaml file with the assertions (a bit similar to how it is done in pytest => e.g. this file).
2. Could it also be possible to add a tag system (similar to pytest --tag). This would make it easier to run the needed tests on Github Actions instead of having to supply the whole file path.

Cheers,
Nicolas

Couldn't import custom library file

I am not able to import custom library file. Is there any easy way to do that?

Feature request: autogenerate assertions

Hi,

Would it be possible to add a function to auto generate assertions for a test?
e.g. we define a test and then run nf-test create-assertions, which would output size() and checksum assertions for example.

Thanks

Bug Report: PluginTest Failure

Hi nf-test, (@lukfor @seppinho )

Having an issue getting mvn install and mvn test to run without errors on the latest version. The new PluginsTest fails due to a missing class property:
groovy.lang.MissingPropertyException: No such property: fasta for class: sun.nio.fs.UnixPath

However, if I run mvn test -Dtest="PluginsTest" the test passes just fine!

I have attached the surefire report for your reference.

Is this the sort of error you have seen before?

Many thanks in advance,
Aaron

<?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" version="3.0" name="com.askimed.nf.test.plugins.PluginsTest" time="23.353" tests="1" errors="0" skipped="0" failures="1">
  <properties>
    <property name="awt.toolkit" value="sun.awt.X11.XToolkit"/>
    <property name="java.specification.version" value="11"/>
    <property name="sun.cpu.isalist" value=""/>
    <property name="sun.jnu.encoding" value="UTF-8"/>
    <property name="java.class.path" value="/home/a.fishman/repos/nf-test/target/test-classes:/home/a.fishman/repos/nf-test/target/classes:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy/3.0.9/groovy-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-ant/3.0.9/groovy-ant-3.0.9.jar:/home/a.fishman/.m2/repository/org/apache/ant/ant/1.10.11/ant-1.10.11.jar:/home/a.fishman/.m2/repository/org/apache/ant/ant-junit/1.10.11/ant-junit-1.10.11.jar:/home/a.fishman/.m2/repository/org/apache/ant/ant-launcher/1.10.11/ant-launcher-1.10.11.jar:/home/a.fishman/.m2/repository/org/apache/ant/ant-antlr/1.10.11/ant-antlr-1.10.11.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-astbuilder/3.0.9/groovy-astbuilder-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-cli-picocli/3.0.9/groovy-cli-picocli-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-console/3.0.9/groovy-console-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-datetime/3.0.9/groovy-datetime-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-docgenerator/3.0.9/groovy-docgenerator-3.0.9.jar:/home/a.fishman/.m2/repository/com/thoughtworks/qdox/qdox/1.12.1/qdox-1.12.1.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-groovydoc/3.0.9/groovy-groovydoc-3.0.9.jar:/home/a.fishman/.m2/repository/com/github/javaparser/javaparser-core/3.23.0/javaparser-core-3.23.0.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-groovysh/3.0.9/groovy-groovysh-3.0.9.jar:/home/a.fishman/.m2/repository/jline/jline/2.14.6/jline-2.14.6.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-jmx/3.0.9/groovy-jmx-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-json/3.0.9/groovy-json-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-jsr223/3.0.9/groovy-jsr223-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-macro/3.0.9/groovy-macro-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-servlet/3.0.9/groovy-servlet-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-sql/3.0.9/groovy-sql-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-swing/3.0.9/groovy-swing-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-templates/3.0.9/groovy-templates-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-test/3.0.9/groovy-test-3.0.9.jar:/home/a.fishman/.m2/repository/junit/junit/4.13.2/junit-4.13.2.jar:/home/a.fishman/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-test-junit5/3.0.9/groovy-test-junit5-3.0.9.jar:/home/a.fishman/.m2/repository/org/junit/platform/junit-platform-launcher/1.7.2/junit-platform-launcher-1.7.2.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-testng/3.0.9/groovy-testng-3.0.9.jar:/home/a.fishman/.m2/repository/org/testng/testng/7.4.0/testng-7.4.0.jar:/home/a.fishman/.m2/repository/com/beust/jcommander/1.78/jcommander-1.78.jar:/home/a.fishman/.m2/repository/org/webjars/jquery/3.5.1/jquery-3.5.1.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-xml/3.0.9/groovy-xml-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-nio/3.0.9/groovy-nio-3.0.9.jar:/home/a.fishman/.m2/repository/org/apache/ivy/ivy/2.5.0/ivy-2.5.0.jar:/home/a.fishman/.m2/repository/info/picocli/picocli/4.6.1/picocli-4.6.1.jar:/home/a.fishman/.m2/repository/com/opencsv/opencsv/5.5.2/opencsv-5.5.2.jar:/home/a.fishman/.m2/repository/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar:/home/a.fishman/.m2/repository/org/apache/commons/commons-text/1.9/commons-text-1.9.jar:/home/a.fishman/.m2/repository/commons-beanutils/commons-beanutils/1.9.4/commons-beanutils-1.9.4.jar:/home/a.fishman/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/home/a.fishman/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/home/a.fishman/.m2/repository/org/apache/commons/commons-collections4/4.4/commons-collections4-4.4.jar:/home/a.fishman/.m2/repository/org/tap4j/tap4j/4.4.2/tap4j-4.4.2.jar:/home/a.fishman/.m2/repository/org/yaml/snakeyaml/1.17/snakeyaml-1.17.jar:/home/a.fishman/.m2/repository/org/junit/jupiter/junit-jupiter/5.9.1/junit-jupiter-5.9.1.jar:/home/a.fishman/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.9.1/junit-jupiter-api-5.9.1.jar:/home/a.fishman/.m2/repository/org/opentest4j/opentest4j/1.2.0/opentest4j-1.2.0.jar:/home/a.fishman/.m2/repository/org/junit/platform/junit-platform-commons/1.9.1/junit-platform-commons-1.9.1.jar:/home/a.fishman/.m2/repository/org/apiguardian/apiguardian-api/1.1.2/apiguardian-api-1.1.2.jar:/home/a.fishman/.m2/repository/org/junit/jupiter/junit-jupiter-params/5.9.1/junit-jupiter-params-5.9.1.jar:/home/a.fishman/.m2/repository/org/junit/jupiter/junit-jupiter-engine/5.9.1/junit-jupiter-engine-5.9.1.jar:/home/a.fishman/.m2/repository/org/junit/platform/junit-platform-engine/1.9.1/junit-platform-engine-1.9.1.jar:"/>
    <property name="java.vm.vendor" value="JetBrains s.r.o."/>
    <property name="sun.arch.data.model" value="64"/>
    <property name="java.vendor.url" value="https://openjdk.java.net/"/>
    <property name="user.timezone" value="UTC"/>
    <property name="java.vm.specification.version" value="11"/>
    <property name="os.name" value="Linux"/>
    <property name="sun.java.launcher" value="SUN_STANDARD"/>
    <property name="user.country" value="GB"/>
    <property name="sun.boot.library.path" value="/home/a.fishman/miniconda3/lib"/>
    <property name="sun.java.command" value="/home/a.fishman/repos/nf-test/target/surefire/surefirebooter10111723401678877503.jar /home/a.fishman/repos/nf-test/target/surefire 2022-11-01T12-01-16_116-jvmRun1 surefire4671726568315671952tmp surefire_011115813422572361704tmp"/>
    <property name="jdk.debug" value="release"/>
    <property name="surefire.test.class.path" value="/home/a.fishman/repos/nf-test/target/test-classes:/home/a.fishman/repos/nf-test/target/classes:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy/3.0.9/groovy-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-ant/3.0.9/groovy-ant-3.0.9.jar:/home/a.fishman/.m2/repository/org/apache/ant/ant/1.10.11/ant-1.10.11.jar:/home/a.fishman/.m2/repository/org/apache/ant/ant-junit/1.10.11/ant-junit-1.10.11.jar:/home/a.fishman/.m2/repository/org/apache/ant/ant-launcher/1.10.11/ant-launcher-1.10.11.jar:/home/a.fishman/.m2/repository/org/apache/ant/ant-antlr/1.10.11/ant-antlr-1.10.11.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-astbuilder/3.0.9/groovy-astbuilder-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-cli-picocli/3.0.9/groovy-cli-picocli-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-console/3.0.9/groovy-console-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-datetime/3.0.9/groovy-datetime-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-docgenerator/3.0.9/groovy-docgenerator-3.0.9.jar:/home/a.fishman/.m2/repository/com/thoughtworks/qdox/qdox/1.12.1/qdox-1.12.1.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-groovydoc/3.0.9/groovy-groovydoc-3.0.9.jar:/home/a.fishman/.m2/repository/com/github/javaparser/javaparser-core/3.23.0/javaparser-core-3.23.0.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-groovysh/3.0.9/groovy-groovysh-3.0.9.jar:/home/a.fishman/.m2/repository/jline/jline/2.14.6/jline-2.14.6.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-jmx/3.0.9/groovy-jmx-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-json/3.0.9/groovy-json-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-jsr223/3.0.9/groovy-jsr223-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-macro/3.0.9/groovy-macro-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-servlet/3.0.9/groovy-servlet-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-sql/3.0.9/groovy-sql-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-swing/3.0.9/groovy-swing-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-templates/3.0.9/groovy-templates-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-test/3.0.9/groovy-test-3.0.9.jar:/home/a.fishman/.m2/repository/junit/junit/4.13.2/junit-4.13.2.jar:/home/a.fishman/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-test-junit5/3.0.9/groovy-test-junit5-3.0.9.jar:/home/a.fishman/.m2/repository/org/junit/platform/junit-platform-launcher/1.7.2/junit-platform-launcher-1.7.2.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-testng/3.0.9/groovy-testng-3.0.9.jar:/home/a.fishman/.m2/repository/org/testng/testng/7.4.0/testng-7.4.0.jar:/home/a.fishman/.m2/repository/com/beust/jcommander/1.78/jcommander-1.78.jar:/home/a.fishman/.m2/repository/org/webjars/jquery/3.5.1/jquery-3.5.1.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-xml/3.0.9/groovy-xml-3.0.9.jar:/home/a.fishman/.m2/repository/org/codehaus/groovy/groovy-nio/3.0.9/groovy-nio-3.0.9.jar:/home/a.fishman/.m2/repository/org/apache/ivy/ivy/2.5.0/ivy-2.5.0.jar:/home/a.fishman/.m2/repository/info/picocli/picocli/4.6.1/picocli-4.6.1.jar:/home/a.fishman/.m2/repository/com/opencsv/opencsv/5.5.2/opencsv-5.5.2.jar:/home/a.fishman/.m2/repository/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar:/home/a.fishman/.m2/repository/org/apache/commons/commons-text/1.9/commons-text-1.9.jar:/home/a.fishman/.m2/repository/commons-beanutils/commons-beanutils/1.9.4/commons-beanutils-1.9.4.jar:/home/a.fishman/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/home/a.fishman/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/home/a.fishman/.m2/repository/org/apache/commons/commons-collections4/4.4/commons-collections4-4.4.jar:/home/a.fishman/.m2/repository/org/tap4j/tap4j/4.4.2/tap4j-4.4.2.jar:/home/a.fishman/.m2/repository/org/yaml/snakeyaml/1.17/snakeyaml-1.17.jar:/home/a.fishman/.m2/repository/org/junit/jupiter/junit-jupiter/5.9.1/junit-jupiter-5.9.1.jar:/home/a.fishman/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.9.1/junit-jupiter-api-5.9.1.jar:/home/a.fishman/.m2/repository/org/opentest4j/opentest4j/1.2.0/opentest4j-1.2.0.jar:/home/a.fishman/.m2/repository/org/junit/platform/junit-platform-commons/1.9.1/junit-platform-commons-1.9.1.jar:/home/a.fishman/.m2/repository/org/apiguardian/apiguardian-api/1.1.2/apiguardian-api-1.1.2.jar:/home/a.fishman/.m2/repository/org/junit/jupiter/junit-jupiter-params/5.9.1/junit-jupiter-params-5.9.1.jar:/home/a.fishman/.m2/repository/org/junit/jupiter/junit-jupiter-engine/5.9.1/junit-jupiter-engine-5.9.1.jar:/home/a.fishman/.m2/repository/org/junit/platform/junit-platform-engine/1.9.1/junit-platform-engine-1.9.1.jar:"/>
    <property name="sun.cpu.endian" value="little"/>
    <property name="user.home" value="/home/a.fishman"/>
    <property name="user.language" value="en"/>
    <property name="java.specification.vendor" value="Oracle Corporation"/>
    <property name="java.version.date" value="2021-10-19"/>
    <property name="java.home" value="/home/a.fishman/miniconda3"/>
    <property name="file.separator" value="/"/>
    <property name="basedir" value="/home/a.fishman/repos/nf-test"/>
    <property name="java.vm.compressedOopsMode" value="Zero based"/>
    <property name="line.separator" value="&#10;"/>
    <property name="java.specification.name" value="Java Platform API Specification"/>
    <property name="java.vm.specification.vendor" value="Oracle Corporation"/>
    <property name="java.awt.graphicsenv" value="sun.awt.X11GraphicsEnvironment"/>
    <property name="surefire.real.class.path" value="/home/a.fishman/repos/nf-test/target/surefire/surefirebooter10111723401678877503.jar"/>
    <property name="sun.management.compiler" value="HotSpot 64-Bit Tiered Compilers"/>
    <property name="java.runtime.version" value="11.0.13+7-b1751.21"/>
    <property name="user.name" value="a.fishman"/>
    <property name="path.separator" value=":"/>
    <property name="os.version" value="3.10.0-957.21.3.el7.x86_64"/>
    <property name="java.runtime.name" value="OpenJDK Runtime Environment"/>
    <property name="file.encoding" value="UTF-8"/>
    <property name="java.vm.name" value="OpenJDK 64-Bit Server VM"/>
    <property name="java.vendor.version" value="JBR-11.0.13.7-1751.21-jcef"/>
    <property name="localRepository" value="/home/a.fishman/.m2/repository"/>
    <property name="java.vendor.url.bug" value="https://bugreport.java.com/bugreport/"/>
    <property name="java.io.tmpdir" value="/tmp"/>
    <property name="java.version" value="11.0.13"/>
    <property name="user.dir" value="/home/a.fishman/repos/nf-test"/>
    <property name="os.arch" value="amd64"/>
    <property name="java.vm.specification.name" value="Java Virtual Machine Specification"/>
    <property name="java.awt.printerjob" value="sun.print.PSPrinterJob"/>
    <property name="sun.os.patch.level" value="unknown"/>
    <property name="java.library.path" value="/opt/clusterware/opt/genders/lib:/opt/clusterware/opt/ruby/lib:/opt/clusterware/opt/gridscheduler/lib/linux-x64:/opt/clusterware/opt/ruby/lib::/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib"/>
    <property name="java.vm.info" value="mixed mode"/>
    <property name="java.vendor" value="JetBrains s.r.o."/>
    <property name="java.vm.version" value="11.0.13+7-b1751.21"/>
    <property name="sun.io.unicode.encoding" value="UnicodeLittle"/>
    <property name="java.class.version" value="55.0"/>
  </properties>
  <testcase name="testScriptInConfig" classname="com.askimed.nf.test.plugins.PluginsTest" time="23.349">
    <failure message="expected: &lt;0&gt; but was: &lt;1&gt;" type="org.opentest4j.AssertionFailedError"><![CDATA[org.opentest4j.AssertionFailedError: expected: <0> but was: <1>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:528)
	at com.askimed.nf.test.plugins.PluginsTest.testScriptInConfig(PluginsTest.java:38)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
	at org.apache.maven.surefire.junitplatform.LazyLauncher.execute(LazyLauncher.java:55)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:223)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:175)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:139)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:456)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:169)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:595)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:581)
]]></failure>
    <system-out><![CDATA[
🚀 nf-test 0.7.0
https://code.askimed.com/nf-test
(c) 2021 - 2022 Lukas Forer and Sebastian Schoenherr

Load .nf-test/plugins/nft-fasta/1.0.0/nft-fasta-1.0.0.jar
Load .nf-test/plugins/nft-fasta-2/2.0.0/nft-fasta-2-2.0.0.jar

Test COPY_FASTA

  Test [7c08cd80] 'Should work with fasta' Hello FASTA
FAILED (4.523s)

  groovy.lang.MissingPropertyException: No such property: fasta for class: sun.nio.fs.UnixPath
  Nextflow stdout:
  
  Nextflow stderr:
  
  

  Test [8bd203a5] 'Should work with fasta.gz' FAILED (4.356s)

  groovy.lang.MissingPropertyException: No such property: fasta for class: sun.nio.fs.UnixPath
  Nextflow stdout:
  
  Nextflow stderr:
  
  

  Test [59927fad] 'Should work with fasta.gz and fasta' FAILED (4.384s)

  groovy.lang.MissingPropertyException: No such property: fasta for class: sun.nio.fs.UnixPath
  Nextflow stdout:
  
  Nextflow stderr:
  
  

  Test [4878b52a] 'Should work with neq' FAILED (4.423s)

  groovy.lang.MissingPropertyException: No such property: fasta for class: sun.nio.fs.UnixPath
  Nextflow stdout:
  
  Nextflow stderr:
  
  

  Test [69cd9010] 'Sample seq1 should be in fasta.1' FAILED (4.325s)

  groovy.lang.MissingPropertyException: No such property: fasta for class: sun.nio.fs.UnixPath
  Nextflow stdout:
  
  Nextflow stderr:
  
  



FAILURE: Executed 5 tests in 22.017s (5 failed)

]]></system-out>
  </testcase>
</testsuite>%         

Running fails with `com/askimed/nf/test/App has been compiled by a more recent version of the Java Runtime`

Hi askimed team

I'm very excited to try out the nf-test tool, but installing with

curl -fsSL https://code.askimed.com/install/nf-test | bash

and then running with

./nf-test

Gives the error

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.UnsupportedClassVersionError: com/askimed/nf/test/App has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:495)

I'm using:

bash-4.2$ java -version
java version "1.8.0_72-ea"
Java(TM) SE Runtime Environment (build 1.8.0_72-ea-b05)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b05, mixed mode)
bash-4.2$ javac -version
javac 1.8.0_72-ea

Which version of java was used to compille nf-test, and is openjdk 1.8 not supported?

Allow multiple workflows in one test file

Hi, thanks for creating this awesome tool! I'm currently in the process of creating an nf-test prototype for the nf-core/modules repository and have come across some small issues and improvement suggestions:

• To test the modules (which are processes) we use a workflow that consists of the module we want to test (and usually some other modules too, to create the input data for the desired module). The original input data is defined like this, which usually means that the when clause in the test file is empty. nf-test throws an error if there is no when clause (java.lang.NullPointerException). Is it possible to make the when clause not a mandatory option when using nextflow_workflow?

• Similar to the previous issue: A nextlow_workflow needs an output, but our testing workflow doesn't actually have any output (we test the files created in an output directory). Is it also possible for emit to be optional when using nextflow_workflow?

• Because a test is defined as a separate workflow in the current tests, it would be helpful if you could add the ability to be able to use multiple workflows in one test file? (similar to how multiple functions can be tested in one file (https://code.askimed.com/nf-test/testcases/nextflow_function/#multiple-functions))

I hope these won't be an issue to add

Cheers,
Nicolas

Feature request - Order-agnostic JSON/YAML object comparison

Hi,

Thanks for developing the nf-test. It has a great potential to be a go-to tool for the whole Nextflow community.

It would be great if we could add an ability to JSON/YAML object comparison with included support of gzipped files. I believe nf-core and Nextflow people do try to push for using JSON a lot with configs etc. It could super useful to have this easy ability to compare JSON files here.

I believe this package could be helpful? - https://github.com/skyscreamer/JSONassert

Change request: parse test params before process mapping

Hi,

I would like to be able to create and define input files within a test rather than store them externally, like so:

nextflow_workflow {

    name "test_passing_file"
    script "trial.nf"
    workflow "trial"

    test("test_passing_file_1") {
        when {
            params {
                someFile = "r1_file.txt"
            }
            workflow {
                """
                def r1Writer = (new File("r1_file.txt")).newWriter()
                r1Writer.writeLine("r1 test file content")
                r1Writer.flush()
                r1Writer.close()
                input[0] = Channel.from("blah")
                """
            }
        }
        then {
            assert workflow.success

            with(workflow.out.trial_out_ch) {
                assert path(get(0)).readLines() == ["modified 'r1 test file content'"]
            }

            new File(params.someFile).delete()
        }
    }
}

However, in the mock.nf script, the workflow block is included before the parameters are parsed, and so I cannot access them:

// process mapping
def input = []

                def r1Writer = (new File("r1_file.txt")).newWriter()
                r1Writer.writeLine("r1 test file content")
                r1Writer.flush()
                r1Writer.close()
                input[0] = Channel.from("blah")

//----

// include test process
include { trial } from '/Users/l.goodsell/test-nf-test/trial.nf'

// define custom rules for JSON that will be generated.
def jsonOutput =
    new JsonGenerator.Options()
        .excludeNulls()  // Do not include fields with value null..
        .addConverter(Path) { value -> value.toString() } // Custom converter for Path. Only filename
        .build()

Would it be ok to swap the order round, so params are parsed first? I'm happy to raise a PR for this change but I wanted to check if it will cause problems first.

Feature request: auto-unpack tar files

Hi,

Would it be possible to add an option or plugin to auto-unpack tar files when staged? This would be handy when staging directories with test data, to be used within nf-test

Thanks
M

Ability to produce TAP output

It would be helpful to have an option to run nf-test with TAP format output. This would enable nf-test to be used alongside additional unit testing systems as a TAP producer.

For example: nf-test test --tap tap.txt example/*.nf.test

Support of easy reading of VCF files

Hi, it would be really useful if nf-test can read VCF files easily to a structural data class. I think something similar to the JSON extension will be sufficient, e.g.:

def mutation_list = path(process.out.out_ch.get(0)).vcf
mutation_list.each { v ->
 assert v.chr == 'chr10'
}

A quick google search returns me this lib (Groovy NGS Utils), which might be useful for the implementation.

Code coverage

Hi there,

Is it possible to obtain a code coverage report for Nextflow scripts using nf-test? If not, would you have any alternative suggestions?

Many thanks in advance!

403 from https://code.askimed.com/install/nf-test/

Probably wasn't ready to get spammed by CI, but it threw a 403 on one of the actions. I can't find it now, but I'll drop an example if I get it again!

Feature Request: Image comparison as an assertion

Hi,

Image comparison comes in handy in some data processing/generation workflows, as nextflow can easily generate an output file, but could be corrupted without the user knowing about it.

Potentially, could we use this - https://romankh3.github.io/image-comparison/ ?

Support of easy reading of FASTA files

Similar to other file formats, it can be useful for nf-test to support FASTA files, so that FASTA files can be compared agnostic to line wrapping width, e.g.:

$ cat fasta.1.txt
> seq1
AGTACGTAGTAGCTGCTGCTACGTGCGCTAGCTAGTACGTCA
CGACGTAGATGCTAGCTGACTCGATGC
> seq2
CGATCGATCGTACGTCGACTGATCGTAGCTACGTCGTACGTA
GCATCGTCAGTTACTGCATGCTCG
$ cat fasta.2.txt
> seq1
AGTACGTAGTAGCTGCTGCTACGTGCGCTAGCTAGTACGTCACGACGTAGATGCTAGCTGACTC
GATGC
> seq2
CGATCGATCGTACGTCGACTGATCGTAGCTACGTCGTACGTAGCATCGTCAGTTACTGCATGCT
CG

# in nf-test
# assume path.fasta returns a map with sequence ids as kyes and sequences as values
assert path("fastq.1.txt").fasta == path("fastq.2.txt").fasta

It'll be more convinient if it can support gzipped FASTA files.

nf-test init should add .nf-test to .gitignore

If the pipeline is in a git repo, then it might make sense to add the default test directory to .gitignore

Feature request: `profile` support in CLI

Hi,

Would you be able to add a --profile flag to the nf-test test CLI?
That way we could define different config profiles depending on which tests we want to run. e.g. local tests vs full scale tests on a cluster.

Config for separate tests

Hi, thanks again for the quick fix yesterday!

While doing some more testing, I ran into the problem that I couldn't really specify any configuration to a specific test since the nextflow.config file specified in nf-test.config is applied to all tests. I then tried to supply the configuration inside the when clause like this:

when {
    process {
         withName: GATK4_COLLECTSVEVIDENCE {
             ext.args = {"--sample-name ${meta.id}"}
         }
    }
}

But I got following error when trying this:

  groovy.lang.MissingMethodException: No signature of method: main_nf$_run_closure1$_closure2.GATK4_COLLECTSVEVIDENCE() is applicable for argument types: (main_nf$_run_closure1$_closure2$_closure4$_closure6$_closure7) values: [main_nf$_run_closure1$_closure2$_closure4$_closure6$_closure7@22ee2d0]

I've also tried inserting the configuration in the mock.nf using the when: workflow clause, but that didn't work either since the mock.nf isn't a configuration file itself.

So my suggestion is to add a possibility to specify a config file in the tests itself or to have a config clause in- or outside of the when clause in which you can specify specific configuration for each test.

Cheers,
Nicolas

Support for `linesGzip` in snapshots

Hi,

Is it possible to add support for linesGzip to snapshots? In some cases md5sums differ and a simple path snapshot is not possible to use. It would be helpful if I could supply a line number or numbers that the snapshot needs to check for.

Suggestion:

assert snapshot(
     workflow,
     path("${outputDir}/file_1").linesGzip[68],
     path("${outputDir}/file_2").linesGzip[1..15]
).match()

Cheers,
Nicolas

Diffing on snapshot failure

Instead of

Found:
  [
      [
          "cd4_REP1_T1.coverage.hist.txt:md5,7cbb473be8d3b32ff2e52fdf4e5d10d2",
          "cd4_REP1_T1.coverage.stats.txt:md5,f1471b61ac17dba283d80e08450c7e55",
          "cd4_REP2_T1.coverage.hist.txt:md5,8d5258a0882494bc4e3f1aa6aa5ed685",
          "cd4_REP2_T1.coverage.stats.txt:md5,3cdb4473211f9da44166ffa6aaa5b602",
          "cd4_REP3_T1.coverage.hist.txt:md5,4a8397c1ea08f35ca7046e3c2c014a08",
          "cd4_REP3_T1.coverage.stats.txt:md5,b88592b469d504a93f2389255ac15f38",
          "cd4_REP4_T1.coverage.hist.txt:md5,d2cbf89e1f753d99bcd03b4e472270dd",
          "cd4_REP4_T1.coverage.stats.txt:md5,4c2b5fe6793aab8560cc87c176ac06ab",
          "jurkat_REP1_T1.coverage.hist.txt:md5,71a893d9d1d55fd47399f6d47b628d6e",
          "jurkat_REP1_T1.coverage.stats.txt:md5,381c69a30099d82066a959deab1a2569",
          "jurkat_REP2_T1.coverage.hist.txt:md5,d124bdadaf2dc5cdd9e3db5a1ab4c588",
          "jurkat_REP2_T1.coverage.stats.txt:md5,b1940346719e4070091f2eef6e508f49"
      ],
      [
          "cd4.saf:md5,06803c8f05a40c3c11b8e49c51dea8d8",
          "jurkat.saf:md5,5ddd9bc4d8d589f4961736c49ee83c11"
      ],
      [
          "cd4_REP1_T1.minus.bedGraph:md5,2a1c34f9d9ef9ff1b9da7874b9e3aaad",
          "cd4_REP1_T1.plus.bedGraph:md5,1509ec3a921e3109c5914e1bcef8cf33",
          "cd4_REP2_T1.minus.bedGraph:md5,c530bc34fa3ec7ac49e88ff65f9c2f92",
          "cd4_REP2_T1.plus.bedGraph:md5,a675141da2874ec08d91591e5ea8242b",
          "cd4_REP3_T1.minus.bedGraph:md5,1c44c165f663d08c869a013b78f051b0",
          "cd4_REP3_T1.plus.bedGraph:md5,a475098ed14da6f3242c4cfdf3abb713",
          "cd4_REP4_T1.minus.bedGraph:md5,b7a3d3c822155dfd8052c67a330fd6d5",
          "cd4_REP4_T1.plus.bedGraph:md5,5623367b681562a2567afc62a8da4151",
          "cd4_filtered.bed:md5,22a41b0aef6da7f4fa0f0644cc60ef32",
          "jurkat_REP1_T1.minus.bedGraph:md5,8d5d9a41df6eb6c56b1bfd3f39dc1fc6",
          "jurkat_REP1_T1.plus.bedGraph:md5,6ed63e5983edaa74fb3965676efdb674",
          "jurkat_REP2_T1.minus.bedGraph:md5,cd83486cb6bb71e6b6e31a69c94d0d85",
          "jurkat_REP2_T1.plus.bedGraph:md5,569e8a25c9cf48d1ef953f3da77c48a1",
          "jurkat_filtered.bed:md5,a491231815327b0d0fab1885b8a8662e"
      ],
      [
          "cd4-group_cd4-transcripts.featureCounts.txt:md5,3bdd3b7047335e64f8628f15f24e6a6f",
          "cd4-group_cd4-transcripts.featureCounts.txt.summary:md5,7a131fe444546d1aad9c0266d3bca6f8",
          "cd4-group_jurkat-transcripts.featureCounts.txt:md5,48ccae667845ee83941336d7a379ac93",
          "cd4-group_jurkat-transcripts.featureCounts.txt.summary:md5,2157b3c4c5efa5842d878d25770653bd",
          "jurkat-group_cd4-transcripts.featureCounts.txt:md5,881272b7053c737a55698e7bcef16b85",
          "jurkat-group_cd4-transcripts.featureCounts.txt.summary:md5,e03801a977101cd3d4893f3d3da055ec",
          "jurkat-group_jurkat-transcripts.featureCounts.txt:md5,54507dca2e47de6d81def6fb09b5eb0f",
          "jurkat-group_jurkat-transcripts.featureCounts.txt.summary:md5,24b00ae7ebc4000fd2a425126630119e",
          "versions.yml:md5,bbcce99a5bfc13d8ff05cf8de9b8a81e"
      ],
      "cd4.bed:md5,ab94918610a560772fdbec591200295f",
      "jurkat.bed:md5,41908723fa423a78476feb2e7627a3a4",
      [
          "cd4_REP1_T1.c_curve.txt:md5,cf4743abdd355595d6ec1fb3f38e66e5",
          "cd4_REP1_T1.command.log:md5,d41d8cd98f00b204e9800998ecf8427e",
          "cd4_REP1_T1.lc_extrap.txt:md5,6d0e7f782f34ee54213c022cbe19df5b",
          "cd4_REP2_T1.c_curve.txt:md5,cf4743abdd355595d6ec1fb3f38e66e5",
          "cd4_REP2_T1.command.log:md5,d41d8cd98f00b204e9800998ecf8427e",
          "cd4_REP2_T1.lc_extrap.txt:md5,6aac6dd78dfd85a18fadd20be7377019",
          "cd4_REP3_T1.c_curve.txt:md5,cf4743abdd355595d6ec1fb3f38e66e5",
          "cd4_REP3_T1.command.log:md5,d41d8cd98f00b204e9800998ecf8427e",
          "cd4_REP3_T1.lc_extrap.txt:md5,02c4b0fc0afd282a402a316d44598173",
          "cd4_REP4_T1.c_curve.txt:md5,cf4743abdd355595d6ec1fb3f38e66e5",
          "cd4_REP4_T1.command.log:md5,d41d8cd98f00b204e9800998ecf8427e",
          "cd4_REP4_T1.lc_extrap.txt:md5,9efbab9d2aee791e7e9cae9237f2c500",
          "jurkat_REP1_T1.c_curve.txt:md5,cf4743abdd355595d6ec1fb3f38e66e5",
          "jurkat_REP1_T1.command.log:md5,d41d8cd98f00b204e9800998ecf8427e",
          "jurkat_REP1_T1.lc_extrap.txt:md5,390386e428ce783b3a373a5f60ee912f",
          "jurkat_REP2_T1.c_curve.txt:md5,cf4743abdd355595d6ec1fb3f38e66e5",
          "jurkat_REP2_T1.command.log:md5,d41d8cd98f00b204e9800998ecf8427e",
          "jurkat_REP2_T1.lc_extrap.txt:md5,cfdda91d83dd381dcefdc754e4b4c54c"
      ],
      [
          "GRCh38_chr21.fa.fai:md5,6ea51e1c1231da5e49212ea9f493d703",
          "cd4_REP1_T1.flagstat:md5,863e2d506d5cc4239af98a5f31bbc906",
          "cd4_REP1_T1.idxstats:md5,b1dd8bcbd23c53c21f0e11082d9315f2",
          "cd4_REP1_T1.sorted.bam:md5,f4dfb2528f78affc85f05c992e4fe972",
          "cd4_REP1_T1.sorted.bam.bai:md5,1dc0fe0c448f2e3dbfda680b5db741cd",
          "cd4_REP1_T1.stats:md5,6b816cfc1fa141449ebc4ea7b5d26991",
          "cd4_REP2_T1.flagstat:md5,0fd86dbf8f799fad49ba471702979bdc",
          "cd4_REP2_T1.idxstats:md5,53204e4c6a9f68664087e4a8123be46a",
          "cd4_REP2_T1.sorted.bam:md5,e5481c842fc40903c142380d6b8825d9",
          "cd4_REP2_T1.sorted.bam.bai:md5,d5391291c961bcddf10c51cd92d5cce9",
          "cd4_REP2_T1.stats:md5,46b475e90054c53dc959a497b705d106",
          "cd4_REP3_T1.flagstat:md5,793c9cadbf80e35dbae3774e9b5bd93f",
          "cd4_REP3_T1.idxstats:md5,42de227ff397013cf288759871b0e938",
          "cd4_REP3_T1.sorted.bam:md5,8d003df0c701abf9bbfc6d0c65a6cca3",
          "cd4_REP3_T1.sorted.bam.bai:md5,e554799626a3999c37dd56512e6338dc",
          "cd4_REP3_T1.stats:md5,44f2302b459c403ff42ab87e1b5bf398",
          "cd4_REP4_T1.flagstat:md5,ed6c609505097f3cc9d9902d69e016f8",
          "cd4_REP4_T1.idxstats:md5,959dc185ae59de4cebe374026711a55e",
          "cd4_REP4_T1.sorted.bam:md5,ceb01aa190383ed673acc312bb0b8425",
          "cd4_REP4_T1.sorted.bam.bai:md5,5fde426d758941f448f728cc1f081371",
          "cd4_REP4_T1.stats:md5,e430e7268fe86ea2ba73e9f8c886d5ec",
          "jurkat_REP1_T1.flagstat:md5,fd5f02b0f02a407447b804b1d80f5421",
          "jurkat_REP1_T1.idxstats:md5,c61af0847c1ad76c06a8de2815975b32",
          "jurkat_REP1_T1.sorted.bam:md5,ba47885e660b991b778aa7885f76a7e0",
          "jurkat_REP1_T1.sorted.bam.bai:md5,b7d970c7114ac530581a96a31e9f31ae",
          "jurkat_REP1_T1.stats:md5,3a49f50d6dea12fd5b06aa00ac9dfc0a",
          "jurkat_REP2_T1.flagstat:md5,86ed47bd41a745ab59de473082c7742d",
          "jurkat_REP2_T1.idxstats:md5,3db8f88c1f836eb5d10f37ca1df81ae9",
          "jurkat_REP2_T1.sorted.bam:md5,84289e709f10f81579c083b0a3341d67",
          "jurkat_REP2_T1.sorted.bam.bai:md5,860e8f78ebddb4874c72420ae5e81fa2",
          "jurkat_REP2_T1.stats:md5,b54ad94f373efafae76141c103a5dd1c"
      ]
  ]

  Expected:
  [
      [
          "cd4_REP1_T1.coverage.hist.txt:md5,7cbb473be8d3b32ff2e52fdf4e5d10d2",
          "cd4_REP1_T1.coverage.stats.txt:md5,f1471b61ac17dba283d80e08450c7e55",
          "cd4_REP2_T1.coverage.hist.txt:md5,8d5258a0882494bc4e3f1aa6aa5ed685",
          "cd4_REP2_T1.coverage.stats.txt:md5,3cdb4473211f9da44166ffa6aaa5b602",
          "cd4_REP3_T1.coverage.hist.txt:md5,4a8397c1ea08f35ca7046e3c2c014a08",
          "cd4_REP3_T1.coverage.stats.txt:md5,b88592b469d504a93f2389255ac15f38",
          "cd4_REP4_T1.coverage.hist.txt:md5,d2cbf89e1f753d99bcd03b4e472270dd",
          "cd4_REP4_T1.coverage.stats.txt:md5,4c2b5fe6793aab8560cc87c176ac06ab",
          "jurkat_REP1_T1.coverage.hist.txt:md5,71a893d9d1d55fd47399f6d47b628d6e",
          "jurkat_REP1_T1.coverage.stats.txt:md5,381c69a30099d82066a959deab1a2569",
          "jurkat_REP2_T1.coverage.hist.txt:md5,d124bdadaf2dc5cdd9e3db5a1ab4c588",
          "jurkat_REP2_T1.coverage.stats.txt:md5,b1940346719e4070091f2eef6e508f49"
      ],
      [
          "cd4.saf:md5,06803c8f05a40c3c11b8e49c51dea8d8",
          "jurkat.saf:md5,5ddd9bc4d8d589f4961736c49ee83c11"
      ],
      [
          "cd4_REP1_T1.minus.bedGraph:md5,2a1c34f9d9ef9ff1b9da7874b9e3aaad",
          "cd4_REP1_T1.plus.bedGraph:md5,1509ec3a921e3109c5914e1bcef8cf33",
          "cd4_REP2_T1.minus.bedGraph:md5,c530bc34fa3ec7ac49e88ff65f9c2f92",
          "cd4_REP2_T1.plus.bedGraph:md5,a675141da2874ec08d91591e5ea8242b",
          "cd4_REP3_T1.minus.bedGraph:md5,1c44c165f663d08c869a013b78f051b0",
          "cd4_REP3_T1.plus.bedGraph:md5,a475098ed14da6f3242c4cfdf3abb713",
          "cd4_REP4_T1.minus.bedGraph:md5,b7a3d3c822155dfd8052c67a330fd6d5",
          "cd4_REP4_T1.plus.bedGraph:md5,5623367b681562a2567afc62a8da4151",
          "cd4_filtered.bed:md5,22a41b0aef6da7f4fa0f0644cc60ef32",
          "jurkat_REP1_T1.minus.bedGraph:md5,8d5d9a41df6eb6c56b1bfd3f39dc1fc6",
          "jurkat_REP1_T1.plus.bedGraph:md5,6ed63e5983edaa74fb3965676efdb674",
          "jurkat_REP2_T1.minus.bedGraph:md5,cd83486cb6bb71e6b6e31a69c94d0d85",
          "jurkat_REP2_T1.plus.bedGraph:md5,569e8a25c9cf48d1ef953f3da77c48a1",
          "jurkat_filtered.bed:md5,a491231815327b0d0fab1885b8a8662e"
      ],
      [
          "cd4-group_cd4-transcripts.featureCounts.txt:md5,8ca04e68865156f1fddebae8649e135f",
          "cd4-group_cd4-transcripts.featureCounts.txt.summary:md5,990709587073dcea4bec6c318b2d4081",
          "cd4-group_jurkat-transcripts.featureCounts.txt:md5,c51b23855a3844367749489bb67341ee",
          "cd4-group_jurkat-transcripts.featureCounts.txt.summary:md5,3528e863ef9af5b6a3a1f5206fcca7f5",
          "jurkat-group_cd4-transcripts.featureCounts.txt:md5,881272b7053c737a55698e7bcef16b85",
          "jurkat-group_cd4-transcripts.featureCounts.txt.summary:md5,e03801a977101cd3d4893f3d3da055ec",
          "jurkat-group_jurkat-transcripts.featureCounts.txt:md5,54507dca2e47de6d81def6fb09b5eb0f",
          "jurkat-group_jurkat-transcripts.featureCounts.txt.summary:md5,24b00ae7ebc4000fd2a425126630119e",
          "versions.yml:md5,bbcce99a5bfc13d8ff05cf8de9b8a81e"
      ],
      "cd4.bed:md5,ab94918610a560772fdbec591200295f",
      "jurkat.bed:md5,41908723fa423a78476feb2e7627a3a4",
      [
          "cd4_REP1_T1.c_curve.txt:md5,cf4743abdd355595d6ec1fb3f38e66e5",
          "cd4_REP1_T1.command.log:md5,d41d8cd98f00b204e9800998ecf8427e",
          "cd4_REP1_T1.lc_extrap.txt:md5,6d0e7f782f34ee54213c022cbe19df5b",
          "cd4_REP2_T1.c_curve.txt:md5,cf4743abdd355595d6ec1fb3f38e66e5",
          "cd4_REP2_T1.command.log:md5,d41d8cd98f00b204e9800998ecf8427e",
          "cd4_REP2_T1.lc_extrap.txt:md5,6aac6dd78dfd85a18fadd20be7377019",
          "cd4_REP3_T1.c_curve.txt:md5,cf4743abdd355595d6ec1fb3f38e66e5",
          "cd4_REP3_T1.command.log:md5,d41d8cd98f00b204e9800998ecf8427e",
          "cd4_REP3_T1.lc_extrap.txt:md5,02c4b0fc0afd282a402a316d44598173",
          "cd4_REP4_T1.c_curve.txt:md5,cf4743abdd355595d6ec1fb3f38e66e5",
          "cd4_REP4_T1.command.log:md5,d41d8cd98f00b204e9800998ecf8427e",
          "cd4_REP4_T1.lc_extrap.txt:md5,9efbab9d2aee791e7e9cae9237f2c500",
          "jurkat_REP1_T1.c_curve.txt:md5,cf4743abdd355595d6ec1fb3f38e66e5",
          "jurkat_REP1_T1.command.log:md5,d41d8cd98f00b204e9800998ecf8427e",
          "jurkat_REP1_T1.lc_extrap.txt:md5,390386e428ce783b3a373a5f60ee912f",
          "jurkat_REP2_T1.c_curve.txt:md5,cf4743abdd355595d6ec1fb3f38e66e5",
          "jurkat_REP2_T1.command.log:md5,d41d8cd98f00b204e9800998ecf8427e",
          "jurkat_REP2_T1.lc_extrap.txt:md5,cfdda91d83dd381dcefdc754e4b4c54c"
      ],
      [
          "GRCh38_chr21.fa.fai:md5,6ea51e1c1231da5e49212ea9f493d703",
          "cd4_REP1_T1.flagstat:md5,863e2d506d5cc4239af98a5f31bbc906",
          "cd4_REP1_T1.idxstats:md5,b1dd8bcbd23c53c21f0e11082d9315f2",
          "cd4_REP1_T1.sorted.bam:md5,f4dfb2528f78affc85f05c992e4fe972",
          "cd4_REP1_T1.sorted.bam.bai:md5,1dc0fe0c448f2e3dbfda680b5db741cd",
          "cd4_REP1_T1.stats:md5,6b816cfc1fa141449ebc4ea7b5d26991",
          "cd4_REP2_T1.flagstat:md5,0fd86dbf8f799fad49ba471702979bdc",
          "cd4_REP2_T1.idxstats:md5,53204e4c6a9f68664087e4a8123be46a",
          "cd4_REP2_T1.sorted.bam:md5,e5481c842fc40903c142380d6b8825d9",
          "cd4_REP2_T1.sorted.bam.bai:md5,d5391291c961bcddf10c51cd92d5cce9",
          "cd4_REP2_T1.stats:md5,46b475e90054c53dc959a497b705d106",
          "cd4_REP3_T1.flagstat:md5,793c9cadbf80e35dbae3774e9b5bd93f",
          "cd4_REP3_T1.idxstats:md5,42de227ff397013cf288759871b0e938",
          "cd4_REP3_T1.sorted.bam:md5,8d003df0c701abf9bbfc6d0c65a6cca3",
          "cd4_REP3_T1.sorted.bam.bai:md5,e554799626a3999c37dd56512e6338dc",
          "cd4_REP3_T1.stats:md5,44f2302b459c403ff42ab87e1b5bf398",
          "cd4_REP4_T1.flagstat:md5,ed6c609505097f3cc9d9902d69e016f8",
          "cd4_REP4_T1.idxstats:md5,959dc185ae59de4cebe374026711a55e",
          "cd4_REP4_T1.sorted.bam:md5,ceb01aa190383ed673acc312bb0b8425",
          "cd4_REP4_T1.sorted.bam.bai:md5,5fde426d758941f448f728cc1f081371",
          "cd4_REP4_T1.stats:md5,e430e7268fe86ea2ba73e9f8c886d5ec",
          "jurkat_REP1_T1.flagstat:md5,fd5f02b0f02a407447b804b1d80f5421",
          "jurkat_REP1_T1.idxstats:md5,c61af0847c1ad76c06a8de2815975b32",
          "jurkat_REP1_T1.sorted.bam:md5,ba47885e660b991b778aa7885f76a7e0",
          "jurkat_REP1_T1.sorted.bam.bai:md5,b7d970c7114ac530581a96a31e9f31ae",
          "jurkat_REP1_T1.stats:md5,3a49f50d6dea12fd5b06aa00ac9dfc0a",
          "jurkat_REP2_T1.flagstat:md5,86ed47bd41a745ab59de473082c7742d",
          "jurkat_REP2_T1.idxstats:md5,3db8f88c1f836eb5d10f37ca1df81ae9",
          "jurkat_REP2_T1.sorted.bam:md5,84289e709f10f81579c083b0a3341d67",
          "jurkat_REP2_T1.sorted.bam.bai:md5,860e8f78ebddb4874c72420ae5e81fa2",
          "jurkat_REP2_T1.stats:md5,b54ad94f373efafae76141c103a5dd1c"
      ]
  ]
  

I'd love if that was tossed in two seperate files and/or a diff was printed instead. Like so

+               "cd4.featureCounts.txt:md5,15936836b3c0cfe78ca021ce559b7bab",
+              "cd4.featureCounts.txt.summary:md5,60dc73a6f4b0215c01a37d7a37212d87",
-               "cd4.featureCounts.txt:md5,1c253ef47c7625aea4d4d61c28fa1832",
-              "cd4.featureCounts.txt.summary:md5,94fafe1c88532354e3ad58cf04dbf5cd",

Access output of processes without named output channel

Nf-test provides a very convenient way to access named output channels of a Nextflow process, e.g.: process.out.my_outCh, but many of our Nextflow processes do not create a named output channel, e.g.: not using emit the script. I wonder if there Is any method to access these outputs for assertions conveniently?

Bug: error when filename is the same as a stated `param`

Hi,

When trying to execute following test (filename is gemini.test):

nextflow_pipeline {

    name "Gemini tests"
    script "main.nf"

    test("Success") {

        when {
            params {
                gemini   = true
            }
        }

        then {
            assert workflow.success
        }
    }
}

I get the following error:

$ nf-test test tests/gemini.test 

🚀 nf-test 0.7.1
https://code.askimed.com/nf-test
(c) 2021 - 2022 Lukas Forer and Sebastian Schoenherr

Error: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
/home/nvnieuwk/Documents/cmgg/nf-cmgg-germline/tests/gemini.test: 12: you tried to assign a value to the class 'gemini'. Do you have a script with this name?
 @ line 12, column 17.
                   gemini   = true
                   ^

1 error

But the test ran perfectly when I changed the filename from gemini.test to gemini_test.test

MissingPropertyException Issue

I generated the process through nf-test generate process main.nf where main.nf has multiple processes and the file I'm working on is called "main.basecalling.nf.test". I have been trying to get the input into this process by using Channel.fromPath() and file() but it always results in the following error:
groovy.lang.MissingPropertyException: No such property: Channel for class: com.askimed.nf.test.lang.TestContext

The way I try to input is:
input[0] = Channel.fromPath("${params.fast5}", checkIfExists:true) where fast5 is a path declared in the params section. This worked for me on a very simple process that used the same tool, Guppy. But it doesn't work here. Another thing I tried was input[0] = file("${params.fast5}") and I received the exact same error messafe.

Individual Process Testing Where You can Use another process' output as input for the next one

So I am grateful for the recent updates to nf-test. I can now get multiple process tests even if they are in the same file. I know there's pipeline tests, which I've ran successfully, but my team prefers multiple process tests specifically. I was wondering if there's a way to access the outputs of other processes like how it would be done on regular nextflow. Example: flye(guppy_ch.fastq). Can the tests emit outputs too? And can those inputs be passed as input?

Feature request: github ci workflow

Hi,

Would you be able to add a "recommended" github CI workflow example to run nf-test on github actions?
Alternatively, would you be able to add a dedicated action?

Thanks
M

Compare tables of output to a specified level of precision

Would be helpful if nf-test was able to compare tables of data (ie output csv/tsv) to desired level of precision.

Bug: cannot import class from jar file in /lib

Hi,

I have a jar file from the built project https://github.com/ssadedin/groovy-ngs-utils

This jar is placed in the ./lib directory reference by my projects nf-test.config. I want to use this jar in my tests.

I would like to use a line like this in the Then {} block like so:

 def vcf = VCF.parse("$baseDir/tests/test_data/NA12879.vcf.gz")

However, when I add this statement:

import gngs.VCF

I recieve the error:

Error: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
/home/alex/vcgs/repipe/src/tests/generate_denovo_rate.nf.test: 1: unable to resolve class gngs.VCF
 @ line 1, column 1.
   import gngs.VCF

However, if I import everything I receive no error:

import gng.*

Then my code in the then block errors:

  groovy.lang.MissingPropertyException: No such property: VCF for class: com.askimed.nf.test.lang.TestContext

I am certain the class VCF exists and gngs.VCF is a valid way to import it.

Please advise

Bug Report: Generating Multiple Process from the same File

I have a file, main.nf, with many processes for a pipeline. I ran:
nf-test generate process main.nf
and I got the following. Notice that main.process and main.fastq.nf.test are one file called "main. process main.fastq.nf.test" but I think should be separate files.

Feature Request: Auto discovery of test files within the repository

Hi,

Would it be possible to implement an auto-discovery test case feature for nf-test? What I would have in mind would be a similar feature what for example pytest provides :

1. If no args provided, then collection would start from any pre-defined in the config testpaths or the current directory.
2. Recurse into directories unless they match pre-defined norecursedirs in the config file or CLI arg
3. Search in the following directories for files that match specific file extension i.e *.nf.test or test*.nf

This might be a material for other discussion whether what is desired convention for naming the test files of nf-test (as listed in the point 3) and I am only suggesting what we could do here.