jirikuncar / invenio

For contribution please check: https://github.com/jirikuncar/invenio/wiki/Contributor-checklist

Invenio README

Invenio is a free software suite enabling you to run your own digital library or document repository on the web. The technology offered by the software covers all aspects of digital library management, from document ingestion through classification, indexing, and curation up to document dissemination. Invenio complies with standards such as the Open Archives Initiative and uses MARC 21 as its underlying bibliographic format. The flexibility and performance of Invenio make it a comprehensive solution for management of document repositories of moderate to large sizes.

Invenio has been originally developed at CERN to run the CERN document server, managing over 1,000,000 bibliographic records in high-energy physics since 2002, covering articles, books, journals, photos, videos, and more. Invenio is nowadays co-developed by an international collaboration comprising institutes such as CERN, DESY, EPFL, FNAL, SLAC and is being used by many more scientific institutions worldwide.

We aim at user friendliness and speed. Among the features are:

• Navigable collection tree
  • Documents organised in collections
  • Regular and virtual collection trees
  • Customisable portal pages for each collection
  • At CERN, over 1,000,000 documents in 700 collections
• Powerful search engine
  • Specially designed indexes to provide fast search speed for repositories of up to 3,000,000 records
  • Customisable simple and advanced search interfaces
  • Combined metadata, fulltext and citation search in one go
  • Results clustering by collection
• Flexible metadata
  • Standard metadata format (MARC)
  • Handling articles, books, theses, photos, videos, museum objects and more
  • Customisable batch import and web submission workflows
  • Customisable output format display and linking rules
• Collaborative features and personalisation
  • User-defined document baskets
  • User-defined email notification alerts
  • Personalised RSS queries
  • Sharing documents of interest in user groups
  • Peer reviewing and group commenting on documents

Invenio is free software licenced under the terms of the GNU General Public Licence (GPL). It is provided on an "as is" basis, in the hope that it will be useful, but without any warranty. There is a possibility to get commercial support in case of interest.

Invenio runs on Unix-like systems and requires MySQL database server and Apache/Python web application server. Please consult the INSTALL file for more information.

Good luck and thanks for choosing Invenio.

Invenio Development Team

Email: [email protected]

IRC: #invenio on irc.freenode.net

Twitter: http://twitter.com/inveniosoftware

Github: http://github.com/inveniosoftware

URL: http://invenio-software.org

invenio's Issues

Create all needed folders automatically upon installation

Some folders need to exist but are not created automatically.

data/
log/
run/
tmp/
tmp-shared/

Currently they need to be added manually. Otherwise errors occur in the code as this folders are expected.

inveniomanage config set fails at dealing with boolean values

What I did:

(pu)$ git log -1
commit 397955af8cafc290b845e888a780a033121ac8a8

(pu)$ inveniomanage config set DEBUG False
(pu)$ cat var/invenio.base-instance/invenio.cfg | grep DEBUG
DEBUG = True

(pu)$ inveniomanage config set KIKOO_LOL False
(pu)$ cat var/invenio.base-instance/invenio.cfg | grep KIKOO_LOL
KIKOO_LOL = u'False'

What happened:

• False was converted to True for the DEBUG key.
• False was converted to u"False" for the KIKOO_LOL key.

What I expected:
Setting any configuration to either True or False should be refected as the corresponding boolean value.

Update header in python files

Many files are missing # -*- coding: utf-8 -*- on the first line.

$ find invenio -iname \*.py -exec grep -H -E -o -c "^# -\*- coding: utf-8 -\*-$" {} \; | grep py:0 | cut -d: -f1

accounts: remove email editing option

• remove /youraccount/edit/WebAccountSettings

Babel: string formating

You may see problems with string formatting:

_('Hello %s') % 'world'

so then replace it with following:

_('Hello %(who)s', who=world)

output_formats: add to registry

formatter/registry.py should include output_formats

RFC: CSS Code Style

Revision	Date	Author
Initial	2014-05-22	@kneczaj
Formatting	2014-05-22	@jirikuncar

Current situation

Looking around there is no guideline about CSS style, which would be nice to have discussed. I mean classes naming scheme, and how to organize the CSS files in Invenio.

For our classes maybe we should set a standard prefix like: 'invenio-'

I had this book recommended by @jmartinm: "Scalable and Modular Architecture for CSS" by Jonathan Snook - maybe it's in our library. It is exactly about organizing of CSS code and some naming conventions.

Proposal

If you have any other propositions comment it here.

See also #237

UnicodeDecodeError when accessing /record/1 while ln=sk

Encountered on Labs, reproducible on localhost.

Also notice that the Exception emails sent to admins seem to be translated in the language used when the exception was originally raised.

visualizations: user generated visualizations

• use JSONAlchemy storage & deposit form
• D3 installation using Bower/Grunt
• editable deposit forms
• evaluate http://rcharts.io/ (suggested by @lnielsen-cern)
• evaluate http://mpld3.github.io/ (suggested by @tiborsimko)

RFC: JavaScript Code Style

Revision	Date	Author
Initial	2014-05-22	@jirikuncar

Idea

It'd be nice to use the standard way for JavaScript (see also #237).

Proposals

• @lnielsen-cern: See previous discussion as well http://invenio-software.org/wiki/Talk/JSFrameworks
• @giokokos https://github.com/airbnb/javascript

Testing

• @greut The first point to enforce regarding JS is no inline JavaScript, no onclick, no href=javascript:. The second would be tests, QUnit is super simple and nice.
  Extra pain points: jslint, jsfmt

CFG_... paths

Minor thought.
Paths in config varibales i.e. 'CFG_TMPDIR' should end with a separator if it is pointing to a dir. Otherwise, this happens

In: mkdtemp(prefix=cfg['CFG_TMPDIR'])
Out: '/home/user/.virtualenvs/pu/var/tmpjxZvUp'

instead of

Out: '/home/user/.virtualenvs/pu/var/tmp/jxZvUp'

CFG_LANGUAGE_LIST_LONG missing from default config

This variable is used in base/footer.html for generating the list of i18n languages, but it is missing from base/config.py.

Database upgrade recipe

Create an upgrade recipe for database upgrade from invenio/master branch.

Apache configuration generation broken for 2.4

This is because:

Order allow,deny
Allow from all

should become

Require all granted

See: http://invenio-software.org/wiki/HowTo/HowToRunOnApache24

database recreate should empty MongoDB

This is not critical, but maybe we should add a way of clearing all used DB systems in inveniomanage.

database recreate should empty MongoDB

This is not critical, but maybe we should add a way of clearing all used DB systems in inveniomanage.

RFC: fixtures loading from inveniomanage

I think it is sometimes helpful to be able to load single modules or classes with fixtures to Invenio database.

I am going to implement this functionality. There is already load_fixtures function in invenio/base/scripts/database.py. I am going to make this function accessible from the command line. The proposed argument list is:

def load_fixtures(packages=[], fixture_classes=[], truncate_tables_first=False)

with no default packages. The function would take all the fixtures form 'fixtures' directory from packages list, and additionally add single fixtures classes from fixture_classes list.

The options from command line perspective would look like below:

option_packages = manager.option('-p', '--packages', dest='packages',
                                 action='append',
                                 default=[],
                                 help='package import name (repeteable)')

option_fixtures = manager.option('-f', '--fixture_classes', dest='fixture_classes',
                                 action='append',
                                 default=[],
                                 help='fixture classes paths to load')

There is also inveniomanage populate option, maybe it makes more sense to modify this command?

Search field rendering correctly

The search field is mostly not rendering correctly. I have the problem in both Firefox and Chrome. It's however a bit random, and as far as I've been able to detect it's due to different CSS rules being applied.

jsonalchemy: ValueError: Unterminated string

Loading the following record ( https://www.dropbox.com/s/xhu6xduisuevzlq/new-upload.xml ) in the demosite and trying to access it (or index it) gives the following the error:

http://pastebin.com/7P4WV1dL

which shows some underlying problem in jsonalchemy

Broken browse button

• Open https://invenio-demo-next.cern.ch/search#simple-search
• Write Ellis to author field
• Click Browse

Documentation using Sphinx

• write installation guide (#170, #188)
• investigate usage of shpinx-apidoc
• section "how to develop an Invenio module" (in pu) @jalavik
• section "how to add an Flask extension" (modify configuration, link to Flask Extension Development)
• deployment section
• rename INSTALL file to INSTALL.rst (#188)

(more to come)
When you have something create a pull request and in message mention (#161)

search: typeahead.js exception when using typeahead v0.10.1

When opening the index page I get:

Uncaught TypeError: Cannot read property 'inputView' of undefined

at invenio/modules/search/static/js/search/typeahead.js, line 306.

Bower dependency tree:

typeahead.js#0.10.0 bower_components/typeahead.js
└── jquery#2.0.3

This issue appeared after the release of typeahead v0.10.0 (2Feb14). If the previous version should be used, maybe we need to specify that in bower.json (currently we allow any version).

Problem with convert registry?

After rebasing to newest pu, we are seeing this error when calling the convert function in BibConvert.xslt_engine.

Seems to be a discovery problem.

Error: Could not convert record: __import__() argument 1 must be string, not None

Traceback (most recent call last):
  File "/home/jlavik/envs/nextstructure/src/nextstructure/invenio/modules/workflows/tasks/marcxml_tasks.py", line 368, in _convert_record
    obj.data = convert(obj.data, stylesheet)
  File "/home/jlavik/envs/nextstructure/src/nextstructure/invenio/legacy/bibconvert/xslt_engine.py", line 288, in convert
    path_to_templates = templates.get(template_filename, '')
  File "/home/jlavik/envs/nextstructure/src/nextstructure/invenio/utils/datastructures.py", line 88, in get
    return self.__getitem__(key)
  File "/home/jlavik/envs/nextstructure/src/nextstructure/invenio/utils/datastructures.py", line 50, in __getitem__
    self._evaluate_function()
  File "/home/jlavik/envs/nextstructure/src/nextstructure/invenio/utils/datastructures.py", line 46, in _evaluate_function
    self._cached_dict = self._function()
  File "/home/jlavik/envs/nextstructure/src/nextstructure/invenio/legacy/bibconvert/registry.py", line 39, in <lambda>
    registry_namespace=converterext)))
  File "/home/jlavik/envs/nextstructure/local/lib/python2.7/site-packages/werkzeug/local.py", line 369, in <lambda>
    __iter__ = lambda x: iter(x._get_current_object())
  File "/home/jlavik/envs/nextstructure/local/lib/python2.7/site-packages/werkzeug/local.py", line 297, in _get_current_object
    return self.__local()
  File "/home/jlavik/envs/nextstructure/local/lib/python2.7/site-packages/Flask_Registry-0.1-py2.7.egg/flask_registry/__init__.py", line 244, in _lookup
    *args, **kwargs
  File "/home/jlavik/envs/nextstructure/local/lib/python2.7/site-packages/Flask_Registry-0.1-py2.7.egg/flask_registry/registries/modulediscovery.py", line 252, in __init__
    self.discover(app=app)
  File "/home/jlavik/envs/nextstructure/local/lib/python2.7/site-packages/Flask_Registry-0.1-py2.7.egg/flask_registry/registries/modulediscovery.py", line 175, in discover
    self._discover_module(pkg)
  File "/home/jlavik/envs/nextstructure/local/lib/python2.7/site-packages/Flask_Registry-0.1-py2.7.egg/flask_registry/registries/pkgresources.py", line 145, in _discover_module
    if resource_isdir(pkg, self.module_name):
  File "/home/jlavik/envs/nextstructure/local/lib/python2.7/site-packages/pkg_resources.py", line 877, in resource_isdir
    return get_provider(package_or_requirement).resource_isdir(
  File "/home/jlavik/envs/nextstructure/local/lib/python2.7/site-packages/pkg_resources.py", line 233, in get_provider
    __import__(moduleOrReq)
TypeError: __import__() argument 1 must be string, not None

Blocking apache alias generation

inveniomanage apache create-config creates a suggested Apache vhost configuration. However, the auto-generated aliases contain this line:

Alias /admin/ /opt/invenio/var/invenio.base-instance/static/admin/

This will proceed to block any URL route under /admin inside the flask application we have right now. How shall it be handled?

I think it is a recent addition, so without having looked to much into it myself, my first questions are:

• what lies under var/invenio.base-instance/static/admin/?
• how can we make the alias work together with the application?

RFC: Invenio new uploader

Idea

Replace the current BibUpload.

Proposal

Create a new uploader keeping this list of ideas about it in mind:

• Work with JSON natively rather than Marc, although the last could be uploaded as well.
• Run tasks in parallel when possible, for example one huge insert coming from one file could generate several insert tasks that could run in parallel.
• Easy to customise depending on the installation needs. For that we thought about creating lightweight workflows (https://github.com/romanchyla/workflow) for each of the common/current modes, so each installation could override the current ones or implement their owns.
• This lightweight workflows should be made out of small monolithic tasks and it would be nice to have a rollback function for each of this functions in case of error in any of them while running the workflow (reverse workflow?)
• Plug the new documents module with a default document storage (same behaviour as BibDocFile).
• Fire signals before and after running any workflow would be a good thing to "connect" the new uploader with other Invenio tools like bibindex, webcoll, etc.

See WIP implementation at #135

Issue with 404 for /fonts/* (glyphicons broken)

Just noticed that we are occasionally getting 404 and glyphicons are not showing:

For example, if I visit http://localhost:4000/yourtags/display/cloud

127.0.0.1 - - [27/May/2014 08:38:33] "GET /yourtags/fonts/glyphicons-halflings-regular.woff HTTP/1.1" 404 -
127.0.0.1 - - [27/May/2014 08:38:33] "GET /yourtags/fonts/glyphicons-halflings-regular.ttf HTTP/1.1" 404 -
127.0.0.1 - - [27/May/2014 08:38:33] "GET /yourtags/fonts/glyphicons-halflings-regular.svg HTTP/1.1" 404 -

I do not get it on http://localhost:4000/yourmessages/display
but I do see it http://localhost:4000/record/comments/subscriptions.

Can any of you reproduce this?

Grunt: extra spaces are not welcome in the command line arguments

$ grunt build  --path=../../var/invenio.base-instance/static -f
Warning: Task " --path=../../var/invenio.base-instance/static" not found. Used --force, continuing.

Collection landing page: search does not keep collection

In any collection landing page, like http://invenio-demo-next.cern.ch/collection/Books

If a search is performed, the c parameter is not kept and the search is done in all collections

script demosite.py - allow package different from invenio_demosite

Functions create and populate have a signature like:

def populate(packages=['invenio_demosite'], default_data=True, files=None,
             job_id=0, extra_info=None):

def create(packages=['invenio_demosite']):

If now we would like to have our own fixtures and demo records loaded with the same commands on inspire we would need to pass the 'packages' to be, for example, 'inspire'

One option is to add a command line option to pass the package.

Another one would be to dynamically get the packages from the registry but in scripts we are working outside of application context so we would have to use something like:

with current_app.test_request_context

Any suggestion on what is the best approach to make demosite.py configurable? I can create a pull request with the changes

inveniomanage cache reset-recstruct broken

First, what is the current role of:

$ inveniomanage cache reset-recstruct

?

Running it now has the problem:

from invenio.modules.editor.models import Bibfmt
ImportError: cannot import name Bibfmt

I fixed the import to:

from invenio.modules.formatter.models import Bibfmt

But then the following exception is raised:

TypeError: __init__() got an unexpected keyword argument 'id_bibrec'

so what is the current role of reset-recstruct? Should it be kept?

global: Python 3.3+ compatibility

• check print statements
• check exception handling
• check metaclasses
• fix intbitset problem inveniosoftware-contrib/intbitset#3
• use different Python library for MySQL

Issues with workflows and registry

Since the recent revamp of adding a workflow registry there are several issues with loading workflows.

The first one I noticed is that we are only loading workflows under *.workflows, without any depositions. This means that "New deposition" under the demosite is not working at the moment.

While I wanted to figure out how to add that, I came across some issues - perhaps it is my lack of experience/stupidity with registries, but here goes:

Let's look at the issue with depositions.

$ inveniomanage shell

In [1]: current_app.extensions["registry"]
Out[1]: <flask_registry.Registry at 0x20b18d0>

In [2]: list(current_app.extensions["registry"])
Out[2]: ['blueprints', 'admin', 'extensions', 'packages', 'restful', 'deposit_types']

Here you see that workflows is not in the default extension list while deposit_types is. Why is that? Anyway, probably not a big issue. However:

In [3]: list(current_app.extensions["registry"]["deposit_types"])
Out[3]: [invenio_demosite.modules.deposit.workflows.article.article]

In [4]: from invenio.modules.deposit.registry import workflows

In [5]: list(workflows)
Out[5]: 
['generic_harvesting_workflow_with_bibsched',
 'anno_workflow',
 'marcxml_workflow',
 'sample_workflow',
 'full_doc_process',
 'generic_harvesting_workflow']

No article in this list of workflows 👎

The only way I have made it work is through using register():

In [6]: workflows.register(current_app.extensions["registry"]["deposit_types"][0])
In [7]: list(workflows)
Out[7]: 
['generic_harvesting_workflow_with_bibsched',
 'anno_workflow',
 'marcxml_workflow',
 'sample_workflow',
 'full_doc_process',
 'article',
 'generic_harvesting_workflow']

We should probably make this automatic though. What would be the best approach?

While experimenting I came across a weird issue with the .discover() function:
(I added print statements in _discover_module() and register())

In [4]: from flask.ext.registry import RegistryError, RegistryProxy
In [5]: from invenio.modules.workflows.registry import WorkflowsRegistry

In [6]: w3 = RegistryProxy('workflows', WorkflowsRegistry, "workflows")
In [7]: w3.discover()
invenio_demosite
invenio.modules.access
register:  'invenio.modules.access.workflows.anno_workflow'
invenio.modules.accounts
invenio.modules.alerts
invenio.modules.apikeys
invenio.modules.authorids
invenio.modules.authorprofiles
invenio.modules.baskets
invenio.modules.bulletin
invenio.modules.circulation
invenio.modules.classifier
invenio.modules.cloudconnector
invenio.modules.comments
invenio.modules.converter
invenio.modules.dashboard
invenio.modules.deposit
invenio.modules.documentation
invenio.modules.documents
invenio.modules.editor
invenio.modules.encoder
invenio.modules.exporter
invenio.modules.formatter
invenio.modules.groups
invenio.modules.indexer
invenio.modules.jsonalchemy
invenio.modules.knowledge
invenio.modules.linkbacks
invenio.modules.matcher
invenio.modules.merger
invenio.modules.messages
invenio.modules.oaiharvester
invenio.modules.oairepository
invenio.modules.oauth2server
invenio.modules.oauthclient
invenio.modules.pidstore
invenio.modules.previewer
invenio.modules.ranker
invenio.modules.records
invenio.modules.redirector
invenio.modules.refextract
invenio.modules.scheduler
invenio.modules.search
invenio.modules.sequencegenerator
invenio.modules.sorter
invenio.modules.statistics
invenio.modules.submit
invenio.modules.sword
invenio.modules.tags
invenio.modules.textminer
invenio.modules.tickets
invenio.modules.upgrader
invenio.modules.uploader
invenio.modules.webhooks
invenio.modules.workflows  # <-- why no workflows?!
invenio_demosite  # <--- we start again
invenio.modules.access
invenio.modules.accounts
invenio.modules.alerts
invenio.modules.apikeys
invenio.modules.authorids
invenio.modules.authorprofiles
invenio.modules.baskets
invenio.modules.bulletin
invenio.modules.circulation
invenio.modules.classifier
invenio.modules.cloudconnector
invenio.modules.comments
invenio.modules.converter
invenio.modules.dashboard
invenio.modules.deposit
invenio.modules.documentation
invenio.modules.documents
invenio.modules.editor
invenio.modules.encoder
invenio.modules.exporter
invenio.modules.formatter
invenio.modules.groups
invenio.modules.indexer
invenio.modules.jsonalchemy
invenio.modules.knowledge
invenio.modules.linkbacks
invenio.modules.matcher
invenio.modules.merger
invenio.modules.messages
invenio.modules.oaiharvester
invenio.modules.oairepository
invenio.modules.oauth2server
invenio.modules.oauthclient
invenio.modules.pidstore
invenio.modules.previewer
invenio.modules.ranker
invenio.modules.records
invenio.modules.redirector
invenio.modules.refextract
invenio.modules.scheduler
invenio.modules.search
invenio.modules.sequencegenerator
invenio.modules.sorter
invenio.modules.statistics
invenio.modules.submit
invenio.modules.sword
invenio.modules.tags
invenio.modules.textminer
invenio.modules.tickets
invenio.modules.upgrader
invenio.modules.uploader
invenio.modules.webhooks
invenio.modules.workflows # <-- Hmm, now workflows were found
register:  invenio.modules.workflows.workflows.full_doc_process'
register:  invenio.modules.workflows.workflows.generic_harvesting_workflow'
register: invenio.modules.workflows.workflows.generic_harvesting_workflow_with_bibsched
register:  invenio.modules.workflows.workflows.marcxml_workflow
register:  invenio.modules.workflows.workflows.sample_workflow
invenio_demosite # <-- We start for a third round!
invenio.modules.access # <-- Here we would normally see an RegistryError  (but I disabled it for testing)
register:  invenio.modules.access.workflows.anno_workflow
invenio.modules.accounts
invenio.modules.alerts
invenio.modules.apikeys
invenio.modules.authorids
invenio.modules.authorprofiles
invenio.modules.baskets
invenio.modules.bulletin
invenio.modules.circulation
invenio.modules.classifier
invenio.modules.cloudconnector
invenio.modules.comments
invenio.modules.converter
invenio.modules.dashboard
invenio.modules.deposit
invenio.modules.documentation
invenio.modules.documents
invenio.modules.editor
invenio.modules.encoder
invenio.modules.exporter
invenio.modules.formatter
invenio.modules.groups
invenio.modules.indexer
invenio.modules.jsonalchemy
invenio.modules.knowledge
invenio.modules.linkbacks
invenio.modules.matcher
invenio.modules.merger
invenio.modules.messages
invenio.modules.oaiharvester
invenio.modules.oairepository
invenio.modules.oauth2server
invenio.modules.oauthclient
invenio.modules.pidstore
invenio.modules.previewer
invenio.modules.ranker
invenio.modules.records
invenio.modules.redirector
invenio.modules.refextract
invenio.modules.scheduler
invenio.modules.search
invenio.modules.sequencegenerator
invenio.modules.sorter
invenio.modules.statistics
invenio.modules.submit
invenio.modules.sword
invenio.modules.tags
invenio.modules.textminer
invenio.modules.tickets
invenio.modules.upgrader
invenio.modules.uploader
invenio.modules.webhooks
invenio.modules.workflows # <-- Now we would get RegistryError as well
register:  invenio.modules.workflows.workflows.full_doc_process'
register:  invenio.modules.workflows.workflows.generic_harvesting_workflow'
register: invenio.modules.workflows.workflows.generic_harvesting_workflow_with_bibsched
register:  invenio.modules.workflows.workflows.marcxml_workflow
register:  invenio.modules.workflows.workflows.sample_workflow

Do the same command again and it only runs one time though the list:

In [8]: w3.discover()
invenio_demosite
invenio.modules.access
register:  invenio.modules.access.workflows.anno_workflow
invenio.modules.accounts
invenio.modules.alerts
invenio.modules.apikeys
invenio.modules.authorids
invenio.modules.authorprofiles
invenio.modules.baskets
invenio.modules.bulletin
invenio.modules.circulation
invenio.modules.classifier
invenio.modules.cloudconnector
invenio.modules.comments
invenio.modules.converter
invenio.modules.dashboard
invenio.modules.deposit
invenio.modules.documentation
invenio.modules.documents
invenio.modules.editor
invenio.modules.encoder
invenio.modules.exporter
invenio.modules.formatter
invenio.modules.groups
invenio.modules.indexer
invenio.modules.jsonalchemy
invenio.modules.knowledge
invenio.modules.linkbacks
invenio.modules.matcher
invenio.modules.merger
invenio.modules.messages
invenio.modules.oaiharvester
invenio.modules.oairepository
invenio.modules.oauth2server
invenio.modules.oauthclient
invenio.modules.pidstore
invenio.modules.previewer
invenio.modules.ranker
invenio.modules.records
invenio.modules.redirector
invenio.modules.refextract
invenio.modules.scheduler
invenio.modules.search
invenio.modules.sequencegenerator
invenio.modules.sorter
invenio.modules.statistics
invenio.modules.submit
invenio.modules.sword
invenio.modules.tags
invenio.modules.textminer
invenio.modules.tickets
invenio.modules.upgrader
invenio.modules.uploader
invenio.modules.webhooks
invenio.modules.workflows
register:  invenio.modules.workflows.workflows.full_doc_process'
register:  invenio.modules.workflows.workflows.generic_harvesting_workflow'
register: invenio.modules.workflows.workflows.generic_harvesting_workflow_with_bibsched
register:  invenio.modules.workflows.workflows.marcxml_workflow
register:  invenio.modules.workflows.workflows.sample_workflow

Any ideas?

workflows: registry exception

I am getting an exception looking up the workflows registry:

In [1]: from invenio.modules.workflows.registry import workflows

In [2]: workflows
Out[2]: <repr(<flask_registry.RegistryProxy at 0x3a6e190>) failed: AttributeError: 'NoneType' object has no attribute 'config'>

Stacktrace:

/home/someone/.virtualenvs/invenio-next/local/lib/python2.7/site-packages/werkzeug/local.pyc in __repr__(self)
    310     def __repr__(self):
    311         try:
--> 312             obj = self._get_current_object()
    313         except RuntimeError:
    314             return '<%s unbound>' % self.__class__.__name__

/home/someone/.virtualenvs/invenio-next/local/lib/python2.7/site-packages/werkzeug/local.pyc in _get_current_object(self)
    295         """
    296         if not hasattr(self.__local, '__release_local__'):
--> 297             return self.__local()
    298         try:
    299             return getattr(self.__local, self.__name__)

/home/someone/.virtualenvs/invenio-next/local/lib/python2.7/site-packages/Flask_Registry-0.1-py2.7.egg/flask_registry/__init__.pyc in _lookup()
    242                 current_app.extensions['registry'][namespace] = \
    243                     registry_class(
--> 244                         *args, **kwargs
    245                     )
    246             return current_app.extensions['registry'][namespace]

/home/someone/src/invenio/invenio/ext/registry/__init__.pyc in __init__(self, *args, **kwargs)
     69         self._valuegetter = kwargs.pop('valuegetter', None)
     70         super(DictModuleAutoDiscoverySubRegistry, self).__init__(
---> 71             *args, **kwargs
     72         )
     73 

/home/someone/.virtualenvs/invenio-next/local/lib/python2.7/site-packages/Flask_Registry-0.1-py2.7.egg/flask_registry/registries/modulediscovery.pyc in __init__(self, module_name, app, registry_namespace, with_setup, silent)
    250         )
    251         self.app = app
--> 252         self.discover(app=app)

/home/someone/.virtualenvs/invenio-next/local/lib/python2.7/site-packages/Flask_Registry-0.1-py2.7.egg/flask_registry/registries/modulediscovery.pyc in discover(self, app)
    173                 continue
    174 
--> 175             self._discover_module(pkg)
    176 
    177     def _discover_module(self, pkg):

/home/someone/src/invenio/invenio/ext/registry/__init__.pyc in _discover_module(self, pkg)
     29         import_str = pkg + '.' + self.module_name
     30 
---> 31         blacklist = self.app.config.get(
     32             '%s_%s_EXCLUDE' % (self.cfg_var_prefix, self.module_name.upper()),
     33             []

AttributeError: 'NoneType' object has no attribute 'config'

Seems to happen after 62f6836. I will see if I can provide a fix.

inveniomanage collect OSError

OSError happens after running inveniomanage collect.

Offending file is:
/home/jlavik/envs/pudemosite/var/invenio.base-instance/static/js/workflows/actions/approval.js

Fix suggested in #240.

See full output here (I added a ipdb statement to print some values):
https://gist.github.com/jalavik/d718d8ae404af25d042b

CC @greut @jirikuncar

div in messages menu

What is the rationale to use a <div> for the for messages menu item (which get enclosed into a <li><a>?

Result

Expected

To fix it, I would have to change its display mode to inline.

Moreover why is there HTML in the Python code? Is there no better solutions?

The code

• https://github.com/jirikuncar/invenio/blob/pu/invenio/base/templates/header_base.html#L43-L59
• https://github.com/jirikuncar/invenio/blob/pu/invenio/modules/messages/views.py#L42-L57

RFC: Indentation in Jinja2 templates

Revision	Date	Author
Initial	2014-05-15	@jirikuncar
2 vs. 4 spaces	2014-05-20	@jirikuncar

Idea

It'd be nice to use the standard way for indentation in Jinja2 templates (or something close to it).

Indentation in python files is check by our Invenio-Kwalitee tool upon every pull request/commit, however there is no standard on how to indent Jinja2 HTML templates.

There are several examples in Jinja2 documentation (http://jinja.pocoo.org/docs/templates/) that are not consistent about using 2 or 4 spaces for indentation:

2 spaces indentation example:

<table>
{%- for row in items|batch(3, '&nbsp;') %}
  <tr>
  {%- for column in row %}
    <td>{{ column }}</td>
  {%- endfor %}
  </tr>
{%- endfor %}
</table>

4 spaces indentation example:

<ul>
{% for group in persons|groupby('gender') %}
    <li>{{ group.grouper }}<ul>
    {% for person in group.list %}
        <li>{{ person.first_name }} {{ person.last_name }}</li>
    {% endfor %}</ul></li>
{% endfor %}
</ul>

Proposal

• 2 or 4 spaces for indentation
• no indentation before for, block

	+	-
2 spaces	deep nesting	readability?
4 spaces	better readability	nesting?

mysql server has gone away

In an installation that is not often used, I am seeing this problem every now and then. After visiting some pages the error disappears.

I have found the following that seems to be exactly the problem:

http://stackoverflow.com/questions/6471549/avoiding-mysql-server-has-gone-away-on-infrequently-used-python-flask-server

Can this be implemented?

PyLD does not work on Python <=2.6

There are a number of issues with PyLD (from requirements.txt) on Python 2.6 and lower. Unfortunately, the PyLD developers are not very keen on fixing them, so I did it myself. In case you want to deploy on such old versions of Python (e.g., SLC) apply the following patch:

diff --git a/requirements.txt b/requirements.txt                                   
index f247762..17092b8 100644                                                      
--- a/requirements.txt                                                             
+++ b/requirements.txt                                                             
@@ -62,4 +62,4 @@ alembic==0.6.2                                                   
 git+https://github.com/lnielsen-cern/setuptools-bower.git#egg=setuptools-bower-0.1
 importlib==1.0.3                                                                  
 pytz                                                                              
-PyLD                                                                              
+git+git://github.com/adrianp/pyld.git@58f7b1545564f55516b968da06f804f9a253919c#egg=pyld-0.4.10

search: putting whole query into parentheses results in empty result

When the whole query is in parentheses the result is empty:

For e.g. 'collection:"Multimedia & Arts"' works whereas '(collection:"Multimedia & Arts")' gives empty result.

The issue is quite old - on old official inspirehep.net page the bug exists too.

travis: timeouted on Python 2.6

• travis tests take more than 50 minutes on Python 2.6

See #240 (comment)

Create fixtures for `collection_field_fieldvalues`

• See https://github.com/inveniosoftware/invenio-demosite/blob/pu/invenio_demosite/democfgdata.sql#L727

Write installation tutorial

Improve INSTALL file and docs/installation.rst for 2.0 release.

upgrader: ensure no auto-generated output on clean database

Auto-generating an upgrade recipe on a newly created database should not add anything new. Currently lots of things are added. I.e.

inveniomanage database init
inveniomanage database create
inveniomanage upgrader create recipe -a ...

Move plupload utilities from the deposit module to base?

Namely:

• PLUploadWidget from deposit/field_widgets.py
• FileUploadField from deposit/fields/file_upload.py

JSONAlchemy: default values in subfields are ignored

Given the following valid Cerberus schema definition:

{'a_dict': {'type': 'dict', 'required': True, 'schema': {'field1': {'type': 'boolean', 'required': True, 'default': lambda: True}}}}

JSONAlchemy will not populate field1 using the declared default value. To get a default a_dict field value you would need to add:

'default': lambda: {'field1': True}

to a_dict's schema.

RFC: javascript, css and bundles

This follows a discussion with @jirikuncar and @lnielsen-cern regarding the state of scripts and stylesheets spread across modules.

Revision	Date	Author
Idea 2	2014-05-26	@greut
Initial	2014-05-14	@greut

Current situation

Invenio is using two jinja extensions css and js that you've encountered in the templates.

{% js url_for("static", filename="js/jquery.js"), "bundle-name" %}
{% css url_for("static", filename="css/bootstrap.css"), "bundle-name" %}

All the file belonging to the same bundle will be concatenated together into one big file. Filters can be applied as well to have a postprocessing on them, usually minification.

They are doing their job but may cause some trouble.

1. if a module or specific templates decides to add its JavaScript file to an existing bundle, then two big files will be created and served depending on which page you're visiting. It's bad as it created code duplication in the user cache and is wasting the server bandwidth.
2. these bundles are created when the template is parsed. If you don't put them at the right place, then your file will only appear at the second request. See #186, #191.

The good side of the current approach is the flexibility is offers. Selecting what is loaded where is great to not impose big chunks of useless extra downloads coming from a particular module to everyone.

Idea 1

It'd be nicer to use the standard way of declaring bundles (or something close to it). In the Python code rather than in the templates. The Bundle could be defined via the registry by each module, aggregated and optimised (dealing with duplication).

The semantic of the js and css calls would then be changed to mean require rather than include.

{% js ["jquery.js", "jquery-ui.js", "invenio.js"] %}
{% css ["invenio.less", "deposit.css"] %}

Invenio will take care of finding the required bundles and include them (respecting the order you've asked for).

Idea 2

Less

LessCSS support imports, moving the complexity of importing to the files themselves and make overriding easier (by using variables and such).

The drawback is that we lose the knowledge of which file has been loaded (or exists) once the bundle is compiled.

# base.css
@import("../bootstrap/bootstrap.less");
@import("variables.less");
@import("grid.less");

Asynchronous Module Definition (AMD)

AMD made popular by Require.js's Why AMD? is used by jQuery and many others to write modular code.

Just like with LessCSS, a file is declaring its dependencies and a compiler is able to aggregate them into one final file. By moving the complexity and the dependencies into the files, we only see the bundles.

// deposit.js
define("deposit", ["utils.js", "templates/form.js"], function(utils, form) {
    return {alert: function() {
        return form({title: utils.capitalize("hello world!")})
    }
);

Usage

jquery = Bundle(["jquery.js", "jquery-datetime-picker.js"], filters="uglify2")
deposit = Bundle(["deposit.js"], filters="requirejs,uglify2")
invenio = Bundle(["invenio.less", "deposit.css"], filters="less,cleancss")
communities = Bundle(["communities/base.css", "communities/zenodo.css"])

assets.register("jquery.js", jquery)
assets.register("deposit.js", deposit)
assets.register("invenio.css", invenio)
assets.register("communities.css", communities)

{% js ["jquery.js", "deposit.js"] %}
{% css ["invenio.css", "communities.css"] %}

Note

You can always include files using plain old HTML, via <script> and <link>.

RFC: redirecter module

Idea from DevFoum related to discussions on the new Pages module.

Use cases for a simple redirecter module:

• Short URLs is handy for e.g. marketing material.
• Static HTML pages that are moved around.

Out of scope:

• Records can be accessible on several URLs via persistent identifiers.
• Making a static page accessible on several URLs.

deposit: javascript update for some form fields doesn't execute correctly

When trying to fill fields based on user input in a DynamicListField or FormField the page needs to be refreshed before showing the new data.

E.g.

for author in rec.get('authors'):
                    form.authors.append_entry(author)

Where rec.get('authors') is a list of authors names and form.authors is a DynamicListField

The issue occurs in invenio/modules/deposit/static/js/deposit/form.js

I have tried to fix this, but I cannot find what is wrong.
@lnielsen-cern has looked at this also.

problem with creating table pidStore

Got this error message on inveniolabs (SLC6, also on local) when doing:
$ inveniomanage database create

On latest pu: SHA-55260519a1a61df9c87bdb9b24cc4688badda8bb

Create fixtures for `collection_field_fieldvalues`

• See https://github.com/inveniosoftware/invenio-demosite/blob/pu/invenio_demosite/democfgdata.sql#L727