Module for depositing record metadata and uploading files.
This is an experimental developer preview release.
- Free software: MIT license
- Documentation: https://invenio-deposit.readthedocs.io/
Invenio module for depositing metadata using workflows.
Home Page: https://invenio-deposit.readthedocs.io
License: MIT License
Module for depositing record metadata and uploading files.
This is an experimental developer preview release.
Should we consider changing some of the names in the module while it's still easy:
invenio-deposit
-> invenio-deposits
(plus all instances of module name usage in directory structures, config)._deposit
-> _internal
(deposit-specific stuff in JSON metadata)pid_type='dep'
-> pid_type='depid'
(invenio_deposit/config.py)Starting from a freshly created virtualenv. I had to add several packages in requirements.txt
and some have to be installed from git:
Moreover the default schema does not work with angular schema form:
[]should be removed for
items`:
"name": "owners",
"owners": {
"title": "Owners",
"description": "Owners of the deposit (a list of user's IDs).",
"type": "array",
"name": "owners",
"items": {
"type": "integer"
}
},
Note: I'm note sure that $ref
works with angular schema form
for _files
.
At the end I still got errors:
http://localhost:5000/deposit/undefined/toto.pdf
404 not found during the file uploadsAs this module is quite complex and the doc is not yet completely written, it would be great to have a working example.
Thanks in advance.
Replace all assert
statement in api.py
with specific exceptions.
--
see https://github.com/inveniosoftware/invenio-deposit/blob/master/invenio_deposit/api.py#L398-L399
MergeConflict as it is raised inside the programmatic API here: https://github.com/inveniosoftware/invenio-deposit/blob/master/invenio_deposit/api.py#L198 should not be a RESTException and should contain information on the merging conflicts from the UnresolvedConflictsException.
Deposit-specific information ("_deposit": {...}
) should be removed from the JSON before those two steps:
https://github.com/inveniosoftware/invenio-deposit/blob/master/invenio_deposit/api.py#L213
https://github.com/inveniosoftware/invenio-deposit/blob/master/invenio_deposit/api.py#L221
A. Do not render templates in tests
B. Install bower dependencies, collect files, and build assets
param
, returns
, raises
, versionadded
)The following cookiecutter change:
inveniosoftware/cookiecutter-invenio-module#98
should be propagated to this Invenio module.
Namely, in run-tests.sh
, the sphinx for doctests is invoked after pytest run:
$ tail -3 ./\{\{\ cookiecutter.project_shortname\ \}\}/run-tests.sh
sphinx-build -qnNW docs docs/_build/html && python setup.py test && sphinx-build -qnNW -b doctest docs docs/_build/doctest
This sometimes led to problems on Travis CI with the second sphinx-build run due
to "disappearing" dependencies after the example application was tested.
A solution that worked for invenio-marc21 (see
inveniosoftware/invenio-marc21#49 (comment))
and that was integrated in cookiecutter (see
inveniosoftware/cookiecutter-invenio-module#98) was to
run doctest execution in pytest, removing the second sphinx-build
invocation.
This both solved Travis CI build failures and simplified test suite execution.
Note that this change may necessitate to amend the code tests etc so that things
would be executed with the Flask application context (see
inveniosoftware/invenio-marc21@09e98fc).
Bucket is creating on publish
even when no files are attached. This should not happen.
Problem:
When publishing a deposit for the first time. If the deposit has no files, it will still get an _files
field added automatically and this field won't be added in the published record.
Why this happens:
Deposit._publish_new
calls Deposit. _process_files
which calls self.files
which calls invenio_records_files. FilesIterator
which modifies the deposit itself by adding an _files
field.
As Deposit._publish_new
had already made a copy of the deposit data, this new field is not added to the published record.
The impact is low but this is still a discrepancy as the published record should be an exact copy of the deposit when published.
Now the question is:
_files
field from self.files
? This in my opinion is the best design as an iterator should not modify anything. (This is similar to invenio-records-files/issues#21 where a getter creates the bucket)_files
field to the published record data
? Simpler solution but kind of ugly as one has to dig in invenio-records-files to understand why it is needed. This might create bugs later when code is refactored (as it always is one day or the other).In Travis, tests with minimal requirements are failing:
Exception occurred: File "/home/plach/.pyenv/versions/3.5.2/envs/inv-dep-3.5.2/lib/python3.5/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
assert not self.context, 'len(context) = %s' % len(self.context)
AssertionError: len(context) = 5
It has already been raised in this issue.
It can be fixed by upgrading Shinx to 1.5.1 in setup.py
.
Why:
The current recommended way to customise any Record API is to create a sublcass. However there are still places in invenio-deposit with explicit references to the Deposit
or Record
API classes instead of using configuration variables.
Who needs it:
B2Share uses JSON Schemas generated on the fly. The current implementation of the Deposit API class uses invenio-jsonschemas. We need to create a subclass of Deposit in order to override this behavior.
Given the current configuration options for a deposit is not possible to have several/different deposit in one site.
I think options like DEPOSIT_DEFAULT_SCHEMAFORM
or DEPOSIT_DEFAULT_JSONSCHEMA
should go together with the rest of the deposit configuration inside DEPOSIT_REST_ENDPOINTS
.
I am not sure about other things, like templates.
@Kennethhole commented on Sun Sep 21 2014
It happens that authors submit new work with different variations of their name. This could be resolved by letting them enter their ORCID and have Invenio auto complete their name.
I think integrating GET orcid_id API could be helpful?
http://support.orcid.org/knowledgebase/articles/116874-orcid-api-guide
Is this desired by more people?
@lnielsen commented on Mon Sep 22 2014
WebSubmit has been superseded by the new deposit module so since this is a feature request I've tagged it under the deposit module.
Autocompletion from ORCID is definitely something we are interested in. I however don't know which would be the best way. The method you suggest starts with the ORCID, then retrieves the name. Alternatively, you could start with the name and auto-complete using the ORCID public profiles data file (https://orcid.org/content/orcid-public-data-file-2013).
As we enable pid_fetcher and pid_minter override in the REST API. It should also be possible in the programatic API.
On POST calls, since request.view_args['pid_value']
is not part of the url (/api/deposit/depositions
), the detected deposit class always falls back to invenio_deposit.api.Deposit
, causing any additional action links added using the invenio_deposit.urils.mark_as_action
decorator to be skipped.
Perhaps adding a pid_type -> deposit_class
mapping as part of a current_records_rest.default_pid_classes
might help to determine the desired default:
deposit_cls = current_records_rest.default_pid_classes.get(pid.pid_type, Deposit)
if 'pid_value' in request.view_args:
deposit_cls = request.view_args['pid_value'].data[1].__class__
Update: This issue leads to incorrect links
generation which is extremely problematic when there are additional actions implemented in a deposit class, and responses have to be consumed by a Javascript UI application. Example:
MyDeposit.custom_action
)POST /api/deposit/depositions
, gets back depid: 1234
and links
with missing custom_action
link.custom_action
, but link is missing.../api/deposit/depositions/1234
), which now contains the missing custom_action
link.Browser are caching GET request for the same url.
An ETag header should be set when requesting a deposit to be able to return back the latest version of the deposit.
And add deprecation warning
and release v1.0.0a10
In invenio-utils
, the Sherpa/Romeo client was recently removed:
In invenio-deposit
, it is still used:
invenio_deposit/autocomplete_utils.py:23:from invenio_utils.sherpa_romeo import SherpaRomeoSearch
invenio_deposit/autocomplete_utils.py:54:def sherpa_romeo_publishers(dummy_form, dummy_field, term, limit=50):
invenio_deposit/autocomplete_utils.py:57: sherpa_romeo = SherpaRomeoSearch()
invenio_deposit/autocomplete_utils.py:58: publishers = sherpa_romeo.search_publisher(term)
invenio_deposit/autocomplete_utils.py:65:def sherpa_romeo_journals(dummy_form, dummy_field, term, limit=50):
invenio_deposit/fields/issn.py:26:# from ..processor_utils import sherpa_romeo_issn_validate
invenio_deposit/fields/issn.py:43: # validators=[sherpa_romeo_issn_validate] # FIXME
invenio_deposit/fields/keywords.py:40: # validators=[sherpa_romeo_journal_validate], #FIXME
invenio_deposit/fields/keywords.py:41: # autocomplete=sherpa_romeo_journals,
invenio_deposit/processor_utils.py:26:from invenio_utils.sherpa_romeo import SherpaRomeoSearch
invenio_deposit/processor_utils.py:150:def sherpa_romeo_issn_process(form, field, submit=False):
invenio_deposit/processor_utils.py:179:def sherpa_romeo_publisher_process(form, field, submit=False, fields=None):
invenio_deposit/processor_utils.py:247:def sherpa_romeo_journal_process(form, field, submit=False, fields=None):
We'd need to either add the missing bits or else to remove this functionality completely.
(P.S. No other Invenio component depends on invenio_utils.sherpa_romeo
.)
Problem:
examples/app-teardown.sh
calls rm <dir> -Rf
which does not work on MacOS.
rm: ./instance: is a directory
rm: -Rf: No such file or directory
...
Solution:
Just put the options before the file name.
Since we don't support cli
commands for deposit, we should remove the empty functions from the cli.py
file as they look confusing.
Problem
Because of this change now all files exceptions are subclasses of RESTException.
RESTException
are managed in a different way from invenio-rest create_api_errorhandler()
.
Because of InvalidOperationError
doesn't have a specific code
set on the class, it inherits 500
from FilesException
.
It means that is raised the 500 error instead of the 403 set here.
Solutions
Which solution make more sense in this case?
403
for the exception InvalidOperationError'status' in kwargs
? (as done for HTTPException
)ping @lnielsen @jirikuncar
After record being published, only deposit one is indexed. Would be nice to index also corresponding record under records-* index.
I just saw that there is no alpha released of invenio-deposit. Would it be possible to release one so that we can install B2Share with pip install without using the devel requirements?
I can make the PR if it is ok.
Thanks.
invenio_deposit/ext.py
.Problem:
In ext.py we have this code:
A @blueprint.record_once
def extend_default_endpoint_prefixes(_):
...
B current_records_rest = app.extensions['invenio-records-rest']
...
C app.register_blueprint(blueprint)
c
calls A
right away. B
depends on invenio-records-rest
being already loaded.
Temporary Solution:
We cannot load the blueprint out of ext.py as create_blueprint
needs the config.
Thus the simplest fix for now is to use @app.before_first_request
.
Problem with this solution:
This however means that one cannot use default_endpoint_prefixes
before the first request.
@lnielsen commented on Wed Apr 30 2014
Originally on 2013-03-07
It appears that this view
is not currently checking if a file is not empty.
We need to add OAuth2 permission control on actions API.
created_by
pid
field to JSON Schema + MappingConsidering all the changes that have been applied in recent versions of invenio-records-rest
regarding indexing after POST/PUT/DELETE requests, the dreaded @index
decorator should be removed.
The most common issue with the @index
decorator has been the following scenario:
Deposit.X
method is called and finishes successfully. Note that the RecordMetadata.version_id
of the deposit has been bumped to N + 1
@index
kicks in (because e.g. Deposit.commit
was called as well), and directly indexes the modified record (as version N + 1
)db.session.commit()
fails, effecitvely rolling back the transaction and setting RecordMetadata.version_id
back to N
)Deposit.X
's changes and version N + 1
. DB record is still in version N
and has the old metadata... ๐กSince this change probably affects a lot of services that might be (unknowingly) relying on this functionality, some joint efforts should be put in order to address the removal and replacement steps.
Problem:
Not all services have their JSON Schemas coming from files or one JSON Schema per type of PID.
However with the last modification enabling "multiple deposits per site" we started tying together endpoints and JSON Schemas.
Who needs it:
B2Share as it has only one endpoint but multiple JSON Schemas, one per community. The last changes should not impact B2Share for now as it does not use the UI, but this is a dangerous direction for us.
Should be resolved in the same fashion as inveniosoftware/invenio-records-rest#152
The package contains examples/app.py
example application, but there is no test
for it in tests/test_example_app.py
. It should be added. See existing examples:
Add support in deposit record API for merging changes into record. The merging
deals with the following situation:
Since the record was updated after the it was loaded into the deposit, the
deposit will be missing the updates from record v2.
Basic algorithm:
Implementation:
merge_with_published()
Add the ability to publish SSE events, which correspond to a particular deposit.
One possible solution is to register an endpoint to each deposit (i.e. /deposits/<depid>/sse
), where a GET request would return an established connection with an SSE channel (having the same name as the deposit URL). Therefore, all handling of access rights concerning the SSE subscriptions will be provided by the read_permission_factory
on the deposit.
@invenio-developers commented on Sun Jun 08 2014
Issue by kneczaj
Friday Jun 06, 2014 at 16:03 GMT
Originally opened as jirikuncar/invenio#302
Hello,
I just saw that the design of the author field which I introduced in bootstrap migration commit was reverted. In my opinion the previous version (first screenshot) is not ideal because of two things:
My design moves also 'move' button to the left to separate it from 'remove' button as the functionality differs much, and misclicking is not welcome here ;) although this is a minor issue, and for sure will get much discussion so I don't insist on it.
I attach both screenshots to discuss on them. Your opinions and other designs are welcome! :)
Design 1 "small buttons" (original):
Design 2 "large buttons" (mine):
Design no. 2 is a fast revert and just presents the general idea, it may need tuning the fields sizes and removing some css glitches.
@lnielsen commented on Thu Jun 12 2014
Here's a better screenshot of design 1 (it's a bug that the buttons are on top of each other):
The reason for the revert was it was the early days of PU-branch and the change to Design 2 was not discussed with me as the initial author and the only production user of it (Design 2 broke functionality on Zenodo). Additionally we were on a tight deadline so I didn't get to discussed the revert with you as it ought to have been.
It's a bit more calm so I'm happy to start a discussion on how we can create an ideal author/affiliation field. I see the button placement only as a minor part of this.
The goal of the author/affiliation field is to make it as easy as possible for a submitter to enter high quality metadata. To that end I believe we need:
I suggest making a survey of other systems to get ideas for how we can best create the author/affiliation field.
Re. design 2: I don't like the having the buttons as bootstrap input groups (it makes the visually to impactful for me), and I don't like that there's no whitespace in-between the fields.
Re. design 1: I agree that you are more likely to accidentally hit the close button. In my experience it has however not been an issue. Also, I believe having both buttons together, make it easier for the user to identify the possible features of the author field (ordering/removal).
@greut commented on Thu Jun 12 2014
Things to consider:
:hover
state or title
to help the user understanding the UIcheck_elasticsearch
in Invenio-Records-Rest
have been renamed to check_search
doc_type
removed with OpenSearch v2A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.