Invenio module for depositing metadata using workflows.
Home Page: https://invenio-deposit.readthedocs.io
License: MIT License
Module for depositing record metadata and uploading files.
This is an experimental developer preview release.
Should we consider changing some of the names in the module while it's still easy:
invenio-deposit -> invenio-deposits (plus all instances of module name usage in directory structures, config)._deposit -> _internal (deposit-specific stuff in JSON metadata)pid_type='dep' -> pid_type='depid' (invenio_deposit/config.py)Starting from a freshly created virtualenv. I had to add several packages in requirements.txt and some have to be installed from git:
Moreover the default schema does not work with angular schema form: []should be removed foritems`:
"name": "owners",
"owners": {
"title": "Owners",
"description": "Owners of the deposit (a list of user's IDs).",
"type": "array",
"name": "owners",
"items": {
"type": "integer"
}
},Note: I'm note sure that $ref works with angular schema form for _files.
At the end I still got errors:
http://localhost:5000/deposit/undefined/toto.pdf 404 not found during the file uploadsAs this module is quite complex and the doc is not yet completely written, it would be great to have a working example.
Thanks in advance.
Replace all assert statement in api.py with specific exceptions.
--
see https://github.com/inveniosoftware/invenio-deposit/blob/master/invenio_deposit/api.py#L398-L399
MergeConflict as it is raised inside the programmatic API here: https://github.com/inveniosoftware/invenio-deposit/blob/master/invenio_deposit/api.py#L198 should not be a RESTException and should contain information on the merging conflicts from the UnresolvedConflictsException.
Deposit-specific information ("_deposit": {...}) should be removed from the JSON before those two steps:
https://github.com/inveniosoftware/invenio-deposit/blob/master/invenio_deposit/api.py#L213
https://github.com/inveniosoftware/invenio-deposit/blob/master/invenio_deposit/api.py#L221
A. Do not render templates in tests
B. Install bower dependencies, collect files, and build assets
param, returns, raises, versionadded)The following cookiecutter change:
inveniosoftware/cookiecutter-invenio-module#98
should be propagated to this Invenio module.
Namely, in run-tests.sh, the sphinx for doctests is invoked after pytest run:
$ tail -3 ./\{\{\ cookiecutter.project_shortname\ \}\}/run-tests.sh
sphinx-build -qnNW docs docs/_build/html && python setup.py test && sphinx-build -qnNW -b doctest docs docs/_build/doctestThis sometimes led to problems on Travis CI with the second sphinx-build run due
to "disappearing" dependencies after the example application was tested.
A solution that worked for invenio-marc21 (see
inveniosoftware/invenio-marc21#49 (comment))
and that was integrated in cookiecutter (see
inveniosoftware/cookiecutter-invenio-module#98) was to
run doctest execution in pytest, removing the second sphinx-build invocation.
This both solved Travis CI build failures and simplified test suite execution.
Note that this change may necessitate to amend the code tests etc so that things
would be executed with the Flask application context (see
inveniosoftware/invenio-marc21@09e98fc).
Bucket is creating on publish even when no files are attached. This should not happen.
Problem:
When publishing a deposit for the first time. If the deposit has no files, it will still get an _files field added automatically and this field won't be added in the published record.
Why this happens:
Deposit._publish_new calls Deposit. _process_files which calls self.files which calls invenio_records_files. FilesIterator which modifies the deposit itself by adding an _files field.
As Deposit._publish_new had already made a copy of the deposit data, this new field is not added to the published record.
The impact is low but this is still a discrepancy as the published record should be an exact copy of the deposit when published.
Now the question is:
_files field from self.files? This in my opinion is the best design as an iterator should not modify anything. (This is similar to invenio-records-files/issues#21 where a getter creates the bucket)_files field to the published record data? Simpler solution but kind of ugly as one has to dig in invenio-records-files to understand why it is needed. This might create bugs later when code is refactored (as it always is one day or the other).In Travis, tests with minimal requirements are failing:
Exception occurred: File "/home/plach/.pyenv/versions/3.5.2/envs/inv-dep-3.5.2/lib/python3.5/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
assert not self.context, 'len(context) = %s' % len(self.context)
AssertionError: len(context) = 5
It has already been raised in this issue.
It can be fixed by upgrading Shinx to 1.5.1 in setup.py.
Why:
The current recommended way to customise any Record API is to create a sublcass. However there are still places in invenio-deposit with explicit references to the Deposit or Record API classes instead of using configuration variables.
Who needs it:
B2Share uses JSON Schemas generated on the fly. The current implementation of the Deposit API class uses invenio-jsonschemas. We need to create a subclass of Deposit in order to override this behavior.
Given the current configuration options for a deposit is not possible to have several/different deposit in one site.
I think options like DEPOSIT_DEFAULT_SCHEMAFORM or DEPOSIT_DEFAULT_JSONSCHEMA should go together with the rest of the deposit configuration inside DEPOSIT_REST_ENDPOINTS.
I am not sure about other things, like templates.
@Kennethhole commented on Sun Sep 21 2014
It happens that authors submit new work with different variations of their name. This could be resolved by letting them enter their ORCID and have Invenio auto complete their name.
I think integrating GET orcid_id API could be helpful?
http://support.orcid.org/knowledgebase/articles/116874-orcid-api-guide
Is this desired by more people?
@lnielsen commented on Mon Sep 22 2014
WebSubmit has been superseded by the new deposit module so since this is a feature request I've tagged it under the deposit module.
Autocompletion from ORCID is definitely something we are interested in. I however don't know which would be the best way. The method you suggest starts with the ORCID, then retrieves the name. Alternatively, you could start with the name and auto-complete using the ORCID public profiles data file (https://orcid.org/content/orcid-public-data-file-2013).
As we enable pid_fetcher and pid_minter override in the REST API. It should also be possible in the programatic API.
On POST calls, since request.view_args['pid_value'] is not part of the url (/api/deposit/depositions), the detected deposit class always falls back to invenio_deposit.api.Deposit, causing any additional action links added using the invenio_deposit.urils.mark_as_action decorator to be skipped.
Perhaps adding a pid_type -> deposit_class mapping as part of a current_records_rest.default_pid_classes might help to determine the desired default:
deposit_cls = current_records_rest.default_pid_classes.get(pid.pid_type, Deposit)
if 'pid_value' in request.view_args:
deposit_cls = request.view_args['pid_value'].data[1].__class__Update: This issue leads to incorrect links generation which is extremely problematic when there are additional actions implemented in a deposit class, and responses have to be consumed by a Javascript UI application. Example:
MyDeposit.custom_action)POST /api/deposit/depositions, gets back depid: 1234 and links with missing custom_action link.custom_action, but link is missing.../api/deposit/depositions/1234), which now contains the missing custom_action link.Browser are caching GET request for the same url.
An ETag header should be set when requesting a deposit to be able to return back the latest version of the deposit.
And add deprecation warning
and release v1.0.0a10
In invenio-utils, the Sherpa/Romeo client was recently removed:
In invenio-deposit, it is still used:
invenio_deposit/autocomplete_utils.py:23:from invenio_utils.sherpa_romeo import SherpaRomeoSearch
invenio_deposit/autocomplete_utils.py:54:def sherpa_romeo_publishers(dummy_form, dummy_field, term, limit=50):
invenio_deposit/autocomplete_utils.py:57: sherpa_romeo = SherpaRomeoSearch()
invenio_deposit/autocomplete_utils.py:58: publishers = sherpa_romeo.search_publisher(term)
invenio_deposit/autocomplete_utils.py:65:def sherpa_romeo_journals(dummy_form, dummy_field, term, limit=50):
invenio_deposit/fields/issn.py:26:# from ..processor_utils import sherpa_romeo_issn_validate
invenio_deposit/fields/issn.py:43: # validators=[sherpa_romeo_issn_validate] # FIXME
invenio_deposit/fields/keywords.py:40: # validators=[sherpa_romeo_journal_validate], #FIXME
invenio_deposit/fields/keywords.py:41: # autocomplete=sherpa_romeo_journals,
invenio_deposit/processor_utils.py:26:from invenio_utils.sherpa_romeo import SherpaRomeoSearch
invenio_deposit/processor_utils.py:150:def sherpa_romeo_issn_process(form, field, submit=False):
invenio_deposit/processor_utils.py:179:def sherpa_romeo_publisher_process(form, field, submit=False, fields=None):
invenio_deposit/processor_utils.py:247:def sherpa_romeo_journal_process(form, field, submit=False, fields=None):
We'd need to either add the missing bits or else to remove this functionality completely.
(P.S. No other Invenio component depends on invenio_utils.sherpa_romeo.)
Problem:
examples/app-teardown.sh calls rm <dir> -Rf which does not work on MacOS.
rm: ./instance: is a directory
rm: -Rf: No such file or directory
...
Solution:
Just put the options before the file name.
Since we don't support cli commands for deposit, we should remove the empty functions from the cli.py file as they look confusing.
Problem
Because of this change now all files exceptions are subclasses of RESTException.
RESTException are managed in a different way from invenio-rest create_api_errorhandler().
Because of InvalidOperationError doesn't have a specific code set on the class, it inherits 500 from FilesException.
It means that is raised the 500 error instead of the 403 set here.
Solutions
Which solution make more sense in this case?
403 for the exception InvalidOperationError'status' in kwargs? (as done for HTTPException)ping @lnielsen @jirikuncar
After record being published, only deposit one is indexed. Would be nice to index also corresponding record under records-* index.
I just saw that there is no alpha released of invenio-deposit. Would it be possible to release one so that we can install B2Share with pip install without using the devel requirements?
I can make the PR if it is ok.
Thanks.
invenio_deposit/ext.py.Problem:
In ext.py we have this code:
A @blueprint.record_once
def extend_default_endpoint_prefixes(_):
...
B current_records_rest = app.extensions['invenio-records-rest']
...
C app.register_blueprint(blueprint)
c calls A right away. B depends on invenio-records-rest being already loaded.
Temporary Solution:
We cannot load the blueprint out of ext.py as create_blueprint needs the config.
Thus the simplest fix for now is to use @app.before_first_request.
Problem with this solution:
This however means that one cannot use default_endpoint_prefixes before the first request.
@lnielsen commented on Wed Apr 30 2014
Originally on 2013-03-07
It appears that this view
is not currently checking if a file is not empty.
We need to add OAuth2 permission control on actions API.
created_bypid field to JSON Schema + MappingConsidering all the changes that have been applied in recent versions of invenio-records-rest regarding indexing after POST/PUT/DELETE requests, the dreaded @index decorator should be removed.
The most common issue with the @index decorator has been the following scenario:
Deposit.X method is called and finishes successfully. Note that the RecordMetadata.version_id of the deposit has been bumped to N + 1@index kicks in (because e.g. Deposit.commit was called as well), and directly indexes the modified record (as version N + 1)db.session.commit() fails, effecitvely rolling back the transaction and setting RecordMetadata.version_id back to N)Deposit.X's changes and version N + 1. DB record is still in version N and has the old metadata... 😡Since this change probably affects a lot of services that might be (unknowingly) relying on this functionality, some joint efforts should be put in order to address the removal and replacement steps.
Problem:
Not all services have their JSON Schemas coming from files or one JSON Schema per type of PID.
However with the last modification enabling "multiple deposits per site" we started tying together endpoints and JSON Schemas.
Who needs it:
B2Share as it has only one endpoint but multiple JSON Schemas, one per community. The last changes should not impact B2Share for now as it does not use the UI, but this is a dangerous direction for us.
Should be resolved in the same fashion as inveniosoftware/invenio-records-rest#152
The package contains examples/app.py example application, but there is no test
for it in tests/test_example_app.py. It should be added. See existing examples:
Add support in deposit record API for merging changes into record. The merging
deals with the following situation:
Since the record was updated after the it was loaded into the deposit, the
deposit will be missing the updates from record v2.
Basic algorithm:
Implementation:
merge_with_published()Add the ability to publish SSE events, which correspond to a particular deposit.
One possible solution is to register an endpoint to each deposit (i.e. /deposits/<depid>/sse), where a GET request would return an established connection with an SSE channel (having the same name as the deposit URL). Therefore, all handling of access rights concerning the SSE subscriptions will be provided by the read_permission_factory on the deposit.
@invenio-developers commented on Sun Jun 08 2014
Friday Jun 06, 2014 at 16:03 GMT
Originally opened as jirikuncar/invenio#302
Hello,
I just saw that the design of the author field which I introduced in bootstrap migration commit was reverted. In my opinion the previous version (first screenshot) is not ideal because of two things:
My design moves also 'move' button to the left to separate it from 'remove' button as the functionality differs much, and misclicking is not welcome here ;) although this is a minor issue, and for sure will get much discussion so I don't insist on it.
I attach both screenshots to discuss on them. Your opinions and other designs are welcome! :)
Design 1 "small buttons" (original):
Design 2 "large buttons" (mine):
Design no. 2 is a fast revert and just presents the general idea, it may need tuning the fields sizes and removing some css glitches.
@lnielsen commented on Thu Jun 12 2014
Here's a better screenshot of design 1 (it's a bug that the buttons are on top of each other):
The reason for the revert was it was the early days of PU-branch and the change to Design 2 was not discussed with me as the initial author and the only production user of it (Design 2 broke functionality on Zenodo). Additionally we were on a tight deadline so I didn't get to discussed the revert with you as it ought to have been.
It's a bit more calm so I'm happy to start a discussion on how we can create an ideal author/affiliation field. I see the button placement only as a minor part of this.
The goal of the author/affiliation field is to make it as easy as possible for a submitter to enter high quality metadata. To that end I believe we need:
I suggest making a survey of other systems to get ideas for how we can best create the author/affiliation field.
Re. design 2: I don't like the having the buttons as bootstrap input groups (it makes the visually to impactful for me), and I don't like that there's no whitespace in-between the fields.
Re. design 1: I agree that you are more likely to accidentally hit the close button. In my experience it has however not been an issue. Also, I believe having both buttons together, make it easier for the user to identify the possible features of the author field (ordering/removal).
@greut commented on Thu Jun 12 2014
Things to consider:
:hover state or title to help the user understanding the UIcheck_elasticsearch in Invenio-Records-Rest have been renamed to check_searchdoc_type removed with OpenSearch v2
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.