yadage / yadage-schemas

schemas for packtivity and yadage for use in cap

yadage-schemas

This package holds JSON schema definitions for preserving individual processing tasks of scientific workflows (referred to "packtivities" since they including information where to find their respective prepackage sofware environments) as well as schemas to define declaratively workflows that orchestrate multiple of these steps using directed acyclic graphs (DAGs)

Workflows defined this way can be read and executed by these packages:

• Packtivity: https://github.com/yadage/packtivity
• Yadage: https://github.com/yadage/yadage

yadage-schemas's Issues

yadage-validate exit codes

We'd like to use yadage-validate in Travis CI recipes. I noticed that yadage always returns exit code 0 even in case of failures.

Example with good YAML:

$ yadage-validate world_population_analysis.yaml 
workflow validates against schema
$ echo $?
0

Example with bad YAML:

$ yadage-validate invalid.yaml 
ERROR:yadage.validator_workflow:validation error
Traceback (most recent call last):
  File "/home/simko/.virtualenvs/yadage/lib/python2.7/site-packages/yadage/validator_workflow.py", line 36, in main
    workflow, toplevel=toplevel, schemadir=schemadir, validate=True)
  File "/home/simko/.virtualenvs/yadage/lib/python2.7/site-packages/yadage/workflow_loader.py", line 5, in workflow
    data = yadageschemas.load(source, toplevel, schema_name, schemadir, validate)
  File "/home/simko/.virtualenvs/yadage/lib/python2.7/site-packages/yadageschemas/__init__.py", line 172, in load
    validator(schema_name,schemadir).validate(data)
  File "/home/simko/.virtualenvs/yadage/lib/python2.7/site-packages/jsonschema/validators.py", line 130, in validate
    raise error
ValidationError: {'step': {'process': {'cmd': 'jupyter nbconvert --output-dir="{outputdir}" world_population_analysis.ipynb', 'process_type': 'string-interpolated-cmd'}, 'publisher': {'publisher_type': 'frompar-pub', 'outputmap': {'outputfile': 'outputfile'}}, 'environment': {u'imagetag': u'latest', 'image': 'reanahub/reana-demo-worldpopulation', u'envscript': u'', 'environment_type': 'docker-encapsulated', u'env': {}, u'resources': []}}, 'scheduler_type': 'singlestep-stage', 'parameters': None} is not valid under any of the given schemas

Failed validating u'oneOf' in schema[u'properties'][u'stages'][u'items'][u'properties'][u'scheduler']:
    {u'oneOf': [{u'$ref': u'scheduler/singlestep-stage-schema.json#'},
                {u'$ref': u'scheduler/multistep-stage-schema.json#'}],
     u'type': u'object'}

On instance[u'stages'][0][u'scheduler']:
    {'parameters': None,
     'scheduler_type': 'singlestep-stage',
     'step': {'environment': {u'env': {},
                              'environment_type': 'docker-encapsulated',
                              u'envscript': u'',
                              'image': 'reanahub/reana-demo-worldpopulation',
                              u'imagetag': u'latest',
                              u'resources': []},
              'process': {'cmd': 'jupyter nbconvert --output-dir="{outputdir}" world_population_analysis.ipynb',
                          'process_type': 'string-interpolated-cmd'},
              'publisher': {'outputmap': {'outputfile': 'outputfile'},
                            'publisher_type': 'frompar-pub'}}}
workflow does not validate against schema
$ echo $?
0

It would be more comfortable if yadage-validate returned 0 when things are OK and 1 otherwise, as is the usual convention with other validating programs.

(moved from yadage/yadage#2)

validation: yadageschemas.load() return value not JSON serializable

>>> import yadageschemas
>>> import json
>>> result = yadageschemas.load('workflow.yml', toplevel='github:reanahub/reana-demo-root6-roofit', schema_name='yadage/workflow-schema',schemadir=yadageschemas.schemadir,validate=True)
>>> json.dumps(result)
*** TypeError: Object of type 'dict' is not JSON serializable

This shouldn't be the case

Use gh-action-pypi-publish v1.7.0+ APIs

In pypa/gh-action-pypi-publish v1.7.0 inputs were changed to use kebab-case. Update to all inputs that were previously snake_case to use the new input format.

• c.f. https://github.com/pypa/gh-action-pypi-publish/releases/tag/v1.7.0

Ambiguous error message `loading error: cannot find URI [...]*workflow.yml*`

The error message thrown here in raw_with_defaults.py is ambiguous in that it appears to get thrown in both of the following two situations:

1. The workflow.yml file actually doesn't exist at the specified location, or the authenticated user doesn't have permission to access it.
2. The workflow.yml file does exist and the authenticated user has permission to access it, but there's some syntax error in the file (eg. indentation) that prevents yadage from using it.

Perhaps a solution could be to have an initial check that the file actually exists and is accessible, and throw a file can't be accessed error in the event that it isn't, and then subsequently check if the file can be read in successfully with data = urlopen(uri).read(), and throw an error like file cannot be read - check yaml syntax if that check fails.

Specify `six` dependency

Coming here from recast-hep/recast-atlas#76, yadageschemas uses six in

yadage-schemas/yadageschemas/dialects/raw_with_defaults.py

Line 118 in 260fbc4

import six.moves.urllib as urllib

but doesn't specify this in its requirements

yadage-schemas/setup.py

Lines 11 to 17 in 260fbc4

	install_requires = [
	'jsonref',
	'pyyaml',
	'requests[security]>=2.9',
	'jsonschema',
	'click',
	],

This creates problems upstream.

Fix schemabase default

I unfortunately do not know what the schemas directory in https://github.com/lukasheinrich/cap-schemas/tree/master/schema was given that this repo has been renamed to https://github.com/yadage/yadage-schemas/ and the redirect fails as there is no schemas/ directory. So this should be fixed.

Originally posted by @matthewfeickert in #44 (comment)

cli: move yadage-validate from yadage to yadage-schemas

Currently yadage-validate lives under yadage repo, it might be better to have it here and then use it from yadage if necessary.

Currently I am using only yadage-schemas for workflow validation and I can not test from cli.

Regression from breaking change in jsonschema v4.10.0

There is a breaking change in jsonschema v4.10.0 that is causing a regression with yadage-schemas v0.10.7. Running the examples of https://gitlab.cern.ch/recast-atlas/examples/helloworld with the local backend without

python -m pip install --upgrade 'recast-atlas[local]' 'jsonschema<=4.9.1'

results in an error for yadage-schemas 0.10.7 of

2023-02-08 02:23:59,617 | packtivity.asyncback |   INFO | configured pool size to 16
2023-02-08 02:23:59,649 | recastatlas.subcomma |  ERROR | caught exception
Traceback (most recent call last):
  File "/home/feickert/.pyenv/versions/recast-helloworld-dev/lib/python3.10/site-packages/recastatlas/backends/local.py", line 21, in run_workflow
    run_workflow(**spec)
  File "/home/feickert/.pyenv/versions/recast-helloworld-dev/lib/python3.10/site-packages/yadage/steering_api.py", line 19, in run_workflow
    with steering_ctx(*args, **kwargs):
  File "/home/feickert/.pyenv/versions/3.10.4/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/feickert/.pyenv/versions/recast-helloworld-dev/lib/python3.10/site-packages/yadage/steering_api.py", line 89, in steering_ctx
    ys = YadageSteering.create(
  File "/home/feickert/.pyenv/versions/recast-helloworld-dev/lib/python3.10/site-packages/yadage/steering_object.py", line 65, in create
    ctrl = creators["local"](**kw)
  File "/home/feickert/.pyenv/versions/recast-helloworld-dev/lib/python3.10/site-packages/yadage/creators.py", line 58, in local_workflows
    workflow_json = workflow_loader.workflow(
  File "/home/feickert/.pyenv/versions/recast-helloworld-dev/lib/python3.10/site-packages/yadage/workflow_loader.py", line 29, in workflow
    data = yadageschemas.load(
  File "/home/feickert/.pyenv/versions/recast-helloworld-dev/lib/python3.10/site-packages/yadageschemas/__init__.py", line 13, in load
    validate_spec(data,validopts)
  File "/home/feickert/.pyenv/versions/recast-helloworld-dev/lib/python3.10/site-packages/yadageschemas/validator.py", line 13, in validate_spec
    return validator(**validopts).validate(data)
  File "/home/feickert/.pyenv/versions/recast-helloworld-dev/lib/python3.10/site-packages/jsonschema/validators.py", line 283, in validate
    raise error
jsonschema.exceptions.ValidationError: ['init'] is not of type 'object'

Failed validating 'type' in schema['properties']['stages']['items']['properties']['dependencies']:
    {'default': {'dependency_type': 'jsonpath_ready', 'expressions': []},
     'oneOf': [{'$ref': 'predicates/jsonpathready-schema.json#'},
               {'$ref': 'predicates/exprfulfilled-schema.json#'}],
     'type': 'object'}

On instance['stages'][0]['dependencies']:
    ['init']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/feickert/.pyenv/versions/recast-helloworld-dev/lib/python3.10/site-packages/recastatlas/subcommands/run.py", line 56, in run
    run_sync(name, spec, backend=backend)
  File "/home/feickert/.pyenv/versions/recast-helloworld-dev/lib/python3.10/site-packages/recastatlas/backends/__init__.py", line 77, in run_sync
    BACKENDS[backend].run_workflow(name, spec)
  File "/home/feickert/.pyenv/versions/recast-helloworld-dev/lib/python3.10/site-packages/recastatlas/backends/local.py", line 23, in run_workflow
    raise FailedRunException
recastatlas.exceptions.FailedRunException
Error: Workflow failed
Exception ignored in: <function Pool.__del__ at 0x7f5dbd4064d0>
Traceback (most recent call last):
  File "/home/feickert/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/pool.py", line 268, in __del__
    self._change_notifier.put(None)
  File "/home/feickert/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/queues.py", line 378, in put
    self._writer.send_bytes(obj)
  File "/home/feickert/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/connection.py", line 205, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/feickert/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/connection.py", line 416, in _send_bytes
    self._send(header + buf)
  File "/home/feickert/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/connection.py", line 373, in _send
    n = write(self._handle, buf)
OSError: [Errno 9] Bad file descriptor