rabix / cwl-ts

Typescript data model for Common Workflow Language

Rabix has grown up!

Reproducible Analyses for Bioinformatics is now a reality!

Please see

• Common Workflow Language (github) for the specfication that the language part of Rabix merged into.
• Rabix Composer for a visual editor for CWL workflows
• This list of executors/runners that can run workflows in CWL on anything from your laptop to your HPC to your cloud.

If you want to see the old documentation for the project please see here

cwl-ts's Issues

Question: does it support v1.2.0-dev4 or v1.2.0 version?

Just curious if this package can properly parse CWL files with cwlVersion: v1.2.0-dev4 or cwlVersion: v1.2.0

Thanks

use schema.org for license, copyright holder

https://github.com/ProteinsWebTeam/ebi-metagenomics-cwl/blob/master/tools/esl-sfetch-index.cwl#L44

Package only compiled .js and .d.ts files in node module

Commit compiled code and package that.

Command line generation

Either as a separate parser or as part of the model, sample command lines should be generated from CLT descriptions.

Validation for Command Line Tools

Example creating workflow programmatically

Please, could you provide an example of how to create a workflow from scratch?

It would be nice if it contained some of these:

• Add workflow's input and output
• Add a couple of steps with its inputs and outputs
• Edit label and doc
• Serialize

Thanks!

Validation for Workflows

build package leads to typescript compiler error

Hi all!

I try to compile my own version of CWL-TS.

I followed readme instructions by running scripts/package.sh.
However it raises many typescript compiler errors.

Has someone been able to build CWL-TS locally ?

Thanks!

Y.

CWL v1.0

CWL v1.0 has now been released! 🎉

• Draft-4 should be renamed to v1
• Mappings should reflect new features

Map the Schema Salad draft-3

More info on http://www.commonwl.org/draft-3/SchemaSalad.html

Error: Can't resolve 'vm'

I created a simple Angular application and imported cwlts/models. When the application is compiled it complains about 'vm' module not present.

It is fixed if I install 'vm' (npm install vm).

Maybe you should add the 'vm' module to the list of dependencies in your package.json.

ERROR in ./node_modules/cwlts/models/helpers/JSExecutor.js
Module not found: Error: Can't resolve 'vm' in '/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/models/helpers'
resolve 'vm' in '/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/models/helpers'
  Parsed request is a module
  using description file: /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/package.json (relative path: ./models/helpers)
    Field 'browser' doesn't contain a valid alias configuration
    resolve as module
      looking for modules in /Users/eduardo/Development/glue/cwlts-angular/
        using description file: /Users/eduardo/Development/glue/cwlts-angular/package.json (relative path: .)
          Field 'browser' doesn't contain a valid alias configuration
          using description file: /Users/eduardo/Development/glue/cwlts-angular/package.json (relative path: ./vm)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/vm doesn't exist
            .ts
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/vm.ts doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/vm.js doesn't exist
            as directory
              /Users/eduardo/Development/glue/cwlts-angular/vm doesn't exist
      /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/models/helpers/node_modules doesn't exist or is not a directory
      /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/models/node_modules doesn't exist or is not a directory
      /Users/eduardo/Development/glue/cwlts-angular/node_modules/node_modules doesn't exist or is not a directory
      /Users/eduardo/Development/glue/node_modules doesn't exist or is not a directory
      /Users/eduardo/Development/node_modules doesn't exist or is not a directory
      /Users/eduardo/node_modules doesn't exist or is not a directory
      /Users/node_modules doesn't exist or is not a directory
      /node_modules doesn't exist or is not a directory
      looking for modules in /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules
        using description file: /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/package.json (relative path: ./node_modules)
          Field 'browser' doesn't contain a valid alias configuration
      looking for modules in /Users/eduardo/Development/glue/cwlts-angular/node_modules
        using description file: /Users/eduardo/Development/glue/cwlts-angular/package.json (relative path: ./node_modules)
          Field 'browser' doesn't contain a valid alias configuration
          using description file: /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/package.json (relative path: ./node_modules/vm)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
          using description file: /Users/eduardo/Development/glue/cwlts-angular/package.json (relative path: ./node_modules/vm)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm doesn't exist
            .ts
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/vm doesn't exist
            .ts
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm.ts doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/vm.ts doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm.js doesn't exist
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/vm.js doesn't exist
            as directory
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm doesn't exist
            as directory
              /Users/eduardo/Development/glue/cwlts-angular/node_modules/vm doesn't exist
[/Users/eduardo/Development/glue/cwlts-angular/vm]
[/Users/eduardo/Development/glue/cwlts-angular/vm.ts]
[/Users/eduardo/Development/glue/cwlts-angular/vm.js]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/models/helpers/node_modules]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/models/node_modules]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/node_modules]
[/Users/eduardo/Development/glue/node_modules]
[/Users/eduardo/Development/node_modules]
[/Users/eduardo/node_modules]
[/Users/node_modules]
[/node_modules]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/vm]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm.ts]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/vm.ts]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/cwlts/node_modules/vm.js]
[/Users/eduardo/Development/glue/cwlts-angular/node_modules/vm.js]
 @ ./node_modules/cwlts/models/helpers/JSExecutor.js 3:9-22
 @ ./node_modules/cwlts/models/helpers/index.js
 @ ./node_modules/cwlts/models/index.js
 @ ./src/app/app.component.ts
 @ ./src/app/app.module.ts
 @ ./src/main.ts
 @ multi ./src/main.ts

Add steps on the fly

cwl-ts/src/models/generic/WorkflowModel.ts

Line 1077 in c8355c5

protected addStepToGraph(step: StepModel, graph: Graph = this.graph) {

Would it be possible to make this method public ?
Then it would be possible to create steps on the fly.

Analyse similar data structures and abstract away their properties in CL tool mapping

V1.0 schemas do not support arrays as maps

CWL v1.0 supports lists being written as maps (common-workflow-language/common-workflow-language#237). JSON schemas for v1.0 don't support this behavior.

Implement reference resolution

This task includes $import and $include directives.

Create interfaces for the Workflow draft-3 specification

We need to create typings according to http://www.commonwl.org/draft-3/Workflow.html

Implement typings for draft-2 and draft-4

Prevent cwl-ts from deleting requirements

cwl-ts deletes workflow requirements and then puts them back. This should be fixed such that cwl-ts only adds requirements, never deletes existing ones.

Missing name field for SchemaDefRequirement types (CWL issue)

mayacoda commented 17 hours ago
SchemaDefRequirement types (InputRecordSchema, InputArraySchema, and InputEnumSchema) should have the name field. It exists in the schema-def example, but not in the documentation.

common-workflow-language/common-workflow-language#216

Generate command line for v1.0 tools

sbg:draft-2 tools can generate their own command line by passing baseCommand, inputs, arguments and stdin/stdout through a series of parsers which resolve each chunk recursively and return a CommandLinePart

How this works can be found in SBDraft2CommandLineToolModel.ts:185:

private generateCommandLineParts(): Promise<CommandLinePart[]> {
    // creates a flat list of inputs and arguments, sorting them accordingly in the process

    // creates a flat list of job values (flattens record fields, for example)
   
    // runs baseCommands through the parser, returns array of Promises

    // runs flattened inputs and arguments through the parser

    // runs streams through the parser

    // joins and returns all Promise arrays, filtering null promises (if job value was null)
}

The same needs to be done with v1.0 tools, keeping in mind any differences in command lines between draft-2 and v1.0.

Also, I'm sure there are bugs in the current command line, for example I don't think some cases where valueFrom is defined are covered properly.

Add timeout on js execution

Expression execution is susceptible to infinite loops. The JSExecutor used by default doesn't timeout, even though there is a timeout set in the options. This might be a bug caused by the nested Promises used to make execution asynchronous.

JSExecutor.ts

 static evaluate(version: CWLVersion, expr: string, job?: any, self?: any): Promise<any> {
        const options = {
            displayErrors: true,
            timeout: 1000
        };
       ...
}

SecondaryFiles, streamable and format should not be assignable to type record and enum (CWL issue)

This is an issue with the CWL (common-workflow-language/common-workflow-language#215) but we can fix it in the typings.

The draft-3 specification states that CommandInputRecordSchema, CommandInputEnumSchema, InputRecordSchema and InputEnumSchema have the fields secondaryFiles, streamable and format. These fields are only possible for type: File or items: File which types record and enum are not. They should be removed from the specification.

Problem with valueFrom field

rabix/composer#472

Constructs like

    valueFrom: O=$(runtime.outdir)/rna_metrics.txt

result in instability - the command line preview will show errors, which disappear and then reappear with edits in the code pane.

@gungorbudak has tracked these to the latest cwl-ts release

Missing dependency in npm package

I get the following message when running ng serve on a project which uses cwl-ts:

ERROR in ./node_modules/cwlts/models/helpers/JSExecutor.js
Module not found: Error: Can't resolve 'vm' in './node_modules\cwlts\models\helpers'

Running npm install vm solves this error.
I think that the vm module needs to be added to package.json

serialize to array of array instead of array

cwl-ts/src/models/d2sb/SBDraft2WorkflowModel.ts

Line 165 in c8355c5

base.inputs = <SBGWorkflowInputParameter[]> this.inputs.map(i => i.serialize());

Hi!
I faced a little bug while serializing CWL workflow.
When inputs type array contains more than one type it is serialize to an array of array like:
[ [ "int", "File" ] ]
However when the model contains only one typem the serialization is ok:
[ "File" ]

Write a descriptive readme and contribution guidelines

Implementing the data model/architecture approach

Construct a graph from a document containing an array of workflows

I would appreciate advice on how to create a workflow model from a document that contains an array of workflows.

There is an example below of this type of document.

When I create the model I can see the customProps.$graph populated. I would like to construct a graph using #main as the entry point.

Thanks for your help!

{
    "cwlVersion": "v1.0", 
    "$graph": [
        {
            "class": "CommandLineTool", 
            "baseCommand": "dbfetch_lwp.pl", 
            "hints": [
                {
                    "dockerPull": "psafont/ebitools:20170602", 
                    "class": "DockerRequirement"
                }
            ], 
            "inputs": [
                {
                    "type": "File", 
                    "doc": "List of accessions to be retrieved as sequences.", 
                    "inputBinding": {
                        "loadContents": true, 
                        "valueFrom": "$(self.contents)", 
                        "position": 3
                    }, 
                    "id": "#dbfetch.cwl/accessions"
                }, 
                {
                    "type": "string", 
                    "doc": "Database to be searched.", 
                    "inputBinding": {
                        "position": 2
                    }, 
                    "default": "uniprot", 
                    "id": "#dbfetch.cwl/database"
                }, 
                {
                    "type": "string", 
                    "doc": "Type of command-line interface.", 
                    "inputBinding": {
                        "position": 1
                    }, 
                    "default": "fetchBatch", 
                    "id": "#dbfetch.cwl/method"
                }, 
                {
                    "type": "string", 
                    "doc": "Format of the output", 
                    "inputBinding": {
                        "position": 4
                    }, 
                    "default": "fasta", 
                    "id": "#dbfetch.cwl/outformat"
                }, 
                {
                    "type": "string", 
                    "doc": "Style of the output", 
                    "inputBinding": {
                        "position": 5
                    }, 
                    "default": "raw", 
                    "id": "#dbfetch.cwl/outstyle"
                }
            ], 
            "outputs": [
                {
                    "type": "stdout", 
                    "id": "#dbfetch.cwl/sequences"
                }
            ], 
            "id": "#dbfetch.cwl"
        }, 
        {
            "class": "CommandLineTool", 
            "baseCommand": "sh", 
            "hints": [
                {
                    "dockerPull": "psafont/ebitools:20170602", 
                    "class": "DockerRequirement"
                }
            ], 
            "inputs": [
                {
                    "type": "File", 
                    "doc": "File with proteins accessions whose sequences are to be retrieved.", 
                    "inputBinding": {
                        "position": 1
                    }, 
                    "id": "#idsFordbfetch.cwl/accessions_file"
                }, 
                {
                    "type": "string", 
                    "inputBinding": {
                        "position": 0
                    }, 
                    "default": "/usr/src/ebitools/idsFordbfetch.sh", 
                    "id": "#idsFordbfetch.cwl/command"
                }, 
                {
                    "type": "string", 
                    "doc": "Maximum number of records to fetch", 
                    "inputBinding": {
                        "position": 2
                    }, 
                    "default": "20", 
                    "id": "#idsFordbfetch.cwl/numberAccessions"
                }
            ], 
            "outputs": [
                {
                    "type": "stdout", 
                    "id": "#idsFordbfetch.cwl/accessions"
                }
            ], 
            "id": "#idsFordbfetch.cwl"
        }, 
        {
            "label": "Retrieve biological records", 
            "doc": "Retrieve biological records by accession identifier", 
            "class": "Workflow", 
            "inputs": [
                {
                    "label": "File with identifiers", 
                    "doc": "File with identifiers", 
                    "type": "File", 
                    "id": "#main/accessions"
                }, 
                {
                    "label": "Number of records to retrieve", 
                    "doc": "Number of records to retrieve starting from the top.", 
                    "type": [
                        "null", 
                        "string"
                    ], 
                    "id": "#main/numberAccessions"
                }
            ], 
            "outputs": [
                {
                    "label": "Biological records", 
                    "doc": "Output file with retrieved biological records", 
                    "type": "File", 
                    "outputSource": "#main/fetch/sequences", 
                    "id": "#main/sequences"
                }
            ], 
            "steps": [
                {
                    "label": "Dbfetch", 
                    "doc": "Retrieve biological records from identifiers", 
                    "run": "#dbfetch.cwl", 
                    "in": [
                        {
                            "source": "#main/format/accessions", 
                            "id": "#main/fetch/accessions"
                        }
                    ], 
                    "out": [
                        "#main/fetch/sequences"
                    ], 
                    "id": "#main/fetch"
                }, 
                {
                    "label": "Get X top identifiers", 
                    "doc": "Filter the X top identiers", 
                    "run": "#idsFordbfetch.cwl", 
                    "in": [
                        {
                            "source": "#main/accessions", 
                            "id": "#main/format/accessions_file"
                        }
                    ], 
                    "out": [
                        "#main/format/accessions"
                    ], 
                    "id": "#main/format"
                }
            ], 
            "id": "#main"
        }
    ]
}