inveniosoftware / invenio-jsonschemas

Invenio Schema Registry

Invenio-JSONSchemas

Invenio module for building and serving JSONSchemas.

Further documentation is available on https://invenio-jsonschemas.readthedocs.io/

invenio-jsonschemas's Issues

jsonschema > 3.2.0 results in TypeError

Package version (if known): 1.1.3

Describe the bug

When sending a request to the records API, validation fails with TypeError: __init__() got an unexpected keyword argument 'types'.
This is due to a change in the jsonschema API with version >=4.0.0.
Downgrading to jsonschema==3.2.0 resolves the issue.

Steps to Reproduce

1. Follow https://github.com/inveniosoftware/training/tree/master/07-data-models-new-field
2. Send the CURL request in Step 5
3. Observe TypeError

Expected behavior

A response as described in the tutorial.

Screenshots (if applicable)

Additional context

Full stacktrace:

127.0.0.1 - - [30/Sep/2021 15:00:38] "POST /api/records/?prettyprint=1 HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/path/to/my/venv/lib/python3.9/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/path/to/my/venv/lib/python3.9/site-packages/werkzeug/middleware/dispatcher.py", line 66, in __call__
    return app(environ, start_response)
  File "/path/to/my/venv/lib/python3.9/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/path/to/my/venv/lib/python3.9/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/path/to/my/venv/lib/python3.9/site-packages/flask_restful/__init__.py", line 271, in error_router
    return original_handler(e)
  File "/path/to/my/venv/lib/python3.9/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/path/to/my/venv/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/path/to/my/venv/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/path/to/my/venv/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/path/to/my/venv/lib/python3.9/site-packages/flask_restful/__init__.py", line 271, in error_router
    return original_handler(e)
  File "/path/to/my/venv/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/path/to/my/venv/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/path/to/my/venv/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/path/to/my/venv/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/path/to/my/venv/lib/python3.9/site-packages/flask/views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "/path/to/my/venv/lib/python3.9/site-packages/invenio_rest/views.py", line 239, in dispatch_request
    result = super(ContentNegotiatedMethodView, self).dispatch_request(
  File "/path/to/my/venv/lib/python3.9/site-packages/flask/views.py", line 163, in dispatch_request
    return meth(*args, **kwargs)
  File "/path/to/my/venv/lib/python3.9/site-packages/invenio_records_rest/views.py", line 428, in need_record_permission_decorator
    return f(self, record=record, *args, **kwargs)
  File "/path/to/my/venv/lib/python3.9/site-packages/invenio_records_rest/views.py", line 695, in post
    record = self.record_class.create(data, id_=record_uuid)
  File "/path/to/my/venv/lib/python3.9/site-packages/invenio_records_files/api.py", line 322, in create
    record = super(Record, cls).create(data, id_=id_, **kwargs)
  File "/path/to/my/venv/lib/python3.9/site-packages/invenio_records/api.py", line 303, in create
    record._validate(
  File "/path/to/my/venv/lib/python3.9/site-packages/invenio_records/api.py", line 183, in _validate
    _records_state.validate(
  File "/path/to/my/venv/lib/python3.9/site-packages/invenio_records/ext.py", line 35, in validate
    return validate(
  File "/path/to/my/venv/lib/python3.9/site-packages/jsonschema/validators.py", line 964, in validate
    validator = cls(schema, *args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'types'

tests: build failure

https://travis-ci.org/inveniosoftware/invenio-jsonschemas/jobs/560026946

views: check API vs UI app

E.g. are schema URLs exposed on both endpoints?

views: resolving on-the-fly schema references

Currently invenio-jsonschemas directly serves the schema from disk, without first resolving $ref to other schemas.

E.g.: see https://qa.inspirehep.net/schemas/records/hep.json which contains unresolved $refs.

It would be actually great to resolve those and have a self-contained schema being served each time.

tests: build error on release requirements

https://travis-ci.org/inveniosoftware/invenio-jsonschemas/jobs/378447544

global: release v1.1.1

tests:build error due to isort

https://travis-ci.org/inveniosoftware/invenio-jsonschemas/jobs/530784076

release: v1.0.0a1

global: migration to github-actions from travis

global: release checklist step 1-3

• Run the release checklist step 1-3.
• E.g. remove GPLv2 and experimental note from the README.rst.

config: default HTTPS for JSONSCHEMAS_URL_SCHEME

Based on #41 (comment) (cc @lnielsen @hachreak)

global: fix build failure

Same as:

• invenio-jsonschemas #87
• invenio-rest #104
• invenio-search #195

https://travis-ci.org/inveniosoftware/invenio-jsonschemas/jobs/656628893

AttributeError: 'Request' object has no attribute 'is_xhr'

../../../virtualenv/python2.7.15/lib/python2.7/site-packages/werkzeug/local.py:347: AttributeError

Fix also deprecation warning in Travis config:
https://travis-ci.org/inveniosoftware/invenio-jsonschemas/jobs/656628893/config

tests: simplify doctest execution

The following cookiecutter change:

inveniosoftware/cookiecutter-invenio-module#98

should be propagated to this Invenio module.

Namely, in run-tests.sh, the sphinx for doctests is invoked after pytest run:

$ tail -3 ./\{\{\ cookiecutter.project_shortname\ \}\}/run-tests.sh
sphinx-build -qnNW docs docs/_build/html && python setup.py test && sphinx-build -qnNW -b doctest docs docs/_build/doctest

This sometimes led to problems on Travis CI with the second sphinx-build run due
to "disappearing" dependencies after the example application was tested.

A solution that worked for invenio-marc21 (see
inveniosoftware/invenio-marc21#49 (comment))
and that was integrated in cookiecutter (see
inveniosoftware/cookiecutter-invenio-module#98) was to
run doctest execution in pytest, removing the second sphinx-build invocation.

This both solved Travis CI build failures and simplified test suite execution.

Note that this change may necessitate to amend the code tests etc so that things
would be executed with the Flask application context (see
inveniosoftware/invenio-marc21@09e98fc).

tests: addition of test_example_app.py

The package contains examples/app.py example application, but there is no test
for it in tests/test_example_app.py. It should be added. See existing examples:

• https://github.com/inveniosoftware/invenio-records/blob/master/tests/test_example_app.py
• https://github.com/inveniosoftware/invenio-marc21/blob/master/tests/test_example_app.py

JSONSchemas registry

As commented in #1 (comment), a (flask-)registry for overlays and extension to define local JSONSchemas is needed so that this module could serve discovered schemas.

docs: add a Warning recommending not to store or advertise invenio-jsonschema URLs

Related to the discussion in inveniosoftware/invenio-records#117

Problem Description:

• records are versioned and can be reverted to a previous version.
• URLs can always change, be it the hostname or the API.

Thus we need to store references to JSON-Schemas as stable URLs. Example: DOI/EPIC/...

Solution:
We should add a warning in the documentation mentioning that we don't recommend storing invenio-jsonschema URLs directly in records. A more stable URI should be used.

global: migrate to new code structure

travis: doctree build fails

Problem:
Deploy works but there is an error in the logs:

running upload_docs
Upload_docs command is deprecated. Use RTD instead.
creating /home/travis/build/inveniosoftware/invenio-jsonschemas/docs/_build/doctrees
Traceback (most recent call last):
  File "setup.py", line 119, in <module>
    'Development Status :: 4 - Beta',
  File "/opt/python/2.7.9/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/opt/python/2.7.9/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/opt/python/2.7.9/lib/python2.7/distutils/dist.py", line 971, in run_command
    cmd_obj.ensure_finalized()
  File "/opt/python/2.7.9/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/setuptools/command/upload_docs.py", line 65, in finalize_options
    self.target_dir = build_sphinx.builder_target_dir
  File "/opt/python/2.7.9/lib/python2.7/distutils/cmd.py", line 105, in __getattr__
    raise AttributeError, attr
AttributeError: builder_target_dir

See https://travis-ci.org/inveniosoftware/invenio-jsonschemas/jobs/238801698

update changelog and travis

• update changelog with v1.0.2 (backport to fix build) #90
• remove python 2

docs: update structure

• introduce new documentation structure inveniosoftware/cookiecutter-invenio-module#56
• document user perspective
• document signals (if not done)
• document example app (if not done)

release 1.0.2

Pin jsonresolver to <3.0.0 to fix the build on Invenio 3.2

ext: new configuration flags required before reading defaults

This #64 introduced new flags which are required before being initialized by invenio-jsonschemas own config file.

The problem seems to be here and here, where the app.config is queried for the new configuration flags, although these are initialized to the supertype's init_config file, here.

docs: usage documentation

TOC:

• Purpose
  • schema registry for invenio (primarily used by records)
• JSON Schema Basics
  • A quick example of a JSON Schema.
• Configuration (in particular JSONSCHEMA_HOST)
• Registering JSON Schemas
• Exposing JSON Schemas
  • /schemas/... (can be enabled/disabled per app)
• Using/loading JSON Schemas
  • Explain how internally we don't make HTTP request to retrieve schemas.
• Composable schemas with JSONRef
  • Explain how schemas can use JSONRef, and we can served fully rendered schemas (because some clients don't deal well with $refs).

Documentation could be linked to Invenio-Records that also have documentation on records and schemas.

Further:

• Fix ReadTheDocs
• Remove experimental warning

docs: remove experimental warning

Remove This is an experimental developer preview release. from README.rst

global: add MAINTAINERS file

There is no MAINTAINERS file in this repository.

resolve_schema: allOf inside allOf is ignored

At https://github.com/inveniosoftware/invenio-jsonschemas/blob/master/invenio_jsonschemas/utils.py#L41-L47 allOf is popped from schema after parts of allOf are merged. If these parts contain another top-level allOf, it is ignored and its content is removed.

A fix might be:

if 'allOf' in schema:
    all_of = schema.pop('allOf')
    for sub_schema in all_of:
        sub_schema.pop('title', None)
        sub_schema.pop('$id', None)
        schema = _merge_dicts(schema, sub_schema)
    schema = traverse(schema)

If you want, I can prepare a pull request.

Do not use local:// URIs

Package version (if known): current

Describe the bug

Invenio RFC 42 decided that Invenio should use local: URIs for schemas, but:

1. This is a non-IANA URI scheme, and we're exposing it to consumers and users of an Invenio system. Invenio should be standards-based wherever possible, and this breaks that principle
2. The implementation assumes that any schema within a jsonschemas directory is actually local, when implementors may want to locally host an external schema (as we do with the GeoJSON Geometry schema this was an untracked file in my clone of invenio-rdm-record)
3. This is breaking my invenio-swh tests when calling RecordService.create(), as that leads to InvenioJSONSchemasState.refresolver_store() asserting that schema.get("$id") == schema_uri but 'http://json-schema.org/draft-07/schema#' != ''local://external/geojson/Geometry.json' — resolved by removing the untracked file

Screenshots (if applicable)

Expected behavior

Invenio should use HTTPS URIs throughout for schemas. To resolve the JSONSCHEMA_HOST issue targetted by the RFC:

• locally-provided schema URIs should have $id properties without URI scheme or host information (e.g. /schema/records/record-v3.0.0.json)
• the resolver should canonicalise to absolute URIs, created by calling urllib.parse.urljoin(BASE_URL, schema["$id"], both when indexing jsonschema/ directories and when finding schemas for records
  • this means that locally-hosted but external schemas (e.g. GeoJSON) have their external URL preserved
  • it also means that the layout of the jsonschemas directory is irrelevant, as it uses the $id over the filesystem path. That said, it would be possible to have the collection process issue warnings if the path doesn't align with the $id for non-absolute URLs
• records that use schemas that start with the base URL should have their schema URL stored as a relative URL without URI scheme or host, so that these URLs are portable across domain changes
• records and schemas can have their $id and $schema properties made absolute on serialization.

ext: add warning when JSONSCHEMAS_HOST is not set or localhost

• https://github.com/inveniosoftware/invenio-jsonschemas/blob/master/invenio_jsonschemas/ext.py#L196

installation: dependency fix for functools

Extracted from #31

Dependencies fix:

ext.py imports functools32 (see https://github.com/inveniosoftware/invenio-jsonschemas/blob/master/invenio_jsonschemas/ext.py#L43-L46)

try:
    from functools import lru_cache
except ImportError:
    from functools32 import lru_cache

But it's not specified in the setup.py as python2.7 dependency like: https://github.com/inveniosoftware/invenio-collections/blob/4c5fd23a4c3fd24dec92f6215179f2098a684ee5/setup.py#L49-L51

set InvenioJSONSchemas._state in init_app

This scenario will fail with the current code:

ext = InvenioJSONSchemas()
ext.init_app(app)
ext.list_schemas()

This happens because InvenioJSONSchemas._state is not set in init_app
https://github.com/inveniosoftware/invenio-jsonschemas/blob/master/invenio_jsonschemas/ext.py#L157