openstate / jodal

Open gov data platform for journalists: all government data — monitor, filter, forward. (FKA jodal)

jodal

jodal is het JOurnalistiek DAshboard Lokaal.

Get started

1. # clone the repo and chdir to there
2. cd backend && cp config.py.example config.py && cp config.yaml.example config.yaml
3. # Edit config.py and config.yaml accordingly to what you want
4. cd ../docker
5. docker-compose up -d
6. cd ..
7. ./setup.sh
8. docker exec -it jodal_backend_1 ./manage.py scrapers locations

In development mode you can run ./bin/dev.sh from the base directory, which will launch the development environment.

To access the local development environment, add the following in /etc/hosts:

127.0.0.1	api.jodal.nl users.jodal.nl www.jodal.nl app.jodal.nl

Then you can go to http://app.jodal.nl preferably in a private window, because of HSTS parameters on the live setup.

Installing FusionAuth

1. Go to http://localhost:9011/
2. Make an admin account
3. Complete the steps on the main DAshboard
4. Make an application (for example 'jodal') 1. Login ap config:
   • Require an API key: Yes
   • Generate Refresh Tokens: No
   • Enable JWT refresh: No
   • Passwordless: yes 2. Authentication tokens: no 3. JWT Enabled: Yes 4. Self serice registration:
   • Enabled: Yes
   • Require password confirmation: Yes
   • Fields: 5. OAuth config:
   • Require authentication: Yes
   • Generate Refresh Tokens: Yes
   • Logout URL: https://app.jodal.nl/
   • Logout behavior: All applications
   • Authorized origins: – Authorized redirects: https://api.jodal.nl/users/simple/callback, http://localhost:8080/api/2/sessions/callback, https://aleph.openstate.eu/api/2/sessions/callback
   • Enabled grants: Authorization Code, Refresh Token 6. SAML:
   • Enabled: No 7. Roles:
5. Generate an api key for use with the jodal application
6. Settings
7. Cors: Enabled
8. copy the client id and secret from the application to backend/config.py
9. copy the api key to backend/config.py
10. restart backend and api container

deployment

Open Overheidsdata uses Fabric for deployment. Run fab deploy.

migrations

Open Overheidsdata uses alembic for migrations

migrate all up to the latest

docker exec jodal_backend_1 alembic upgrade head

rollback

docker exec jodal_backend_1 alembic downgrade -1

create a migration

docker exec jodal_backend_1 alembic revision -m "create account table"

adding data

Open Overheidsdata runs several scrapers, in the jodal_backend_1 container. Run the floowing steps to get started:

1. docker exec jodal_backend_1 ./mana ge.py scrapers locations
2. docker exec jodal_backend_1 ./mana ge.py scrapers openspending -f 2021-01-01
3. docker exec jodal_backend_1 ./mana ge.py scrapers poliflw -f 2021-01-01
4. docker exec jodal_backend_1 ./mana ge.py scrapers obv -f 2021-01-01

contact

Send an email to [email protected]

jodal's Issues

check if query term is used/pciekd up by openspending fetcher

Include MediaObjects in openbesluitvorming queries

Currently queries the start_date field, but that is not used in the MediaObject documents. Instead the field date_modified is used. So we need to account for that.

Shorten titles toa max amount of characters or something

enable search in document text on openbesluitvorming

Make it possible to have different document types in the 'documents' index in ES

so we can filter on source

Highlights should be highlighted in expanded document view as well.

This can probably be done by removing the highlight tags from the ES response and replacing that string in the full text with the highlighted string that contain the match

sqlalchemy upgrade needs flask sqlalchmeny upgrade too

Not upgrading resulted in this error:

api-jodal_1      | 2021-04-30 10:16:13,329.329.9117088317871:werkzeug:140127739356928:INFO:1:172.18.0.8 - - [30/Apr/2021 10:16:13] "GET /users/simple/me HTTP/1.0" 200 -
api-jodal_1      | 2021-04-30 10:16:13,334.334.5463275909424:app:140127730456320:ERROR:1:Exception on /columns [GET]
api-jodal_1      | Traceback (most recent call last):
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/_collections.py", line 1008, in __call__
api-jodal_1      |     return self.registry[key]
api-jodal_1      | KeyError: <greenlet.greenlet object at 0x7f720679ca90 (otid=0x7f7206859580) current active started main>
api-jodal_1      | 
api-jodal_1      | During handling of the above exception, another exception occurred:
api-jodal_1      | 
api-jodal_1      | Traceback (most recent call last):
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
api-jodal_1      |     rv = self.dispatch_request()
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
api-jodal_1      |     return self.view_functions[rule.endpoint](**req.view_args)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask_restful/__init__.py", line 468, in wrapper
api-jodal_1      |     resp = resource(*args, **kwargs)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask/views.py", line 89, in view
api-jodal_1      |     return self.dispatch_request(*args, **kwargs)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask_restful/__init__.py", line 583, in dispatch_request
api-jodal_1      |     resp = meth(*args, **kwargs)
api-jodal_1      |   File "/opt/jodal/app/resources.py", line 24, in wrapper
api-jodal_1      |     return func(*args, **kwargs)
api-jodal_1      |   File "/opt/jodal/app/resources.py", line 36, in get
api-jodal_1      |     columns = Column.query.filter(Column.user_id==user_id)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask_sqlalchemy/__init__.py", line 514, in __get__
api-jodal_1      |     return type.query_class(mapper, session=self.sa.session())
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 129, in __call__
api-jodal_1      |     return self.registry()
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/_collections.py", line 1010, in __call__
api-jodal_1      |     return self.registry.setdefault(key, self.createfunc())
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 4058, in __call__
api-jodal_1      |     return self.class_(**local_kw)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask_sqlalchemy/__init__.py", line 138, in __init__
api-jodal_1      |     bind = options.pop('bind', None) or db.engine
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask_sqlalchemy/__init__.py", line 943, in engine
api-jodal_1      |     return self.get_engine()
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask_sqlalchemy/__init__.py", line 962, in get_engine
api-jodal_1      |     return connector.get_engine()
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask_sqlalchemy/__init__.py", line 555, in get_engine
api-jodal_1      |     options = self.get_options(sa_url, echo)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask_sqlalchemy/__init__.py", line 570, in get_options
api-jodal_1      |     self._sa.apply_driver_hacks(self._app, sa_url, options)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask_sqlalchemy/__init__.py", line 884, in apply_driver_hacks
api-jodal_1      |     sa_url.query.setdefault('charset', 'utf8')
api-jodal_1      | AttributeError: 'sqlalchemy.cimmutabledict.immutabledict' object has no attribute 'setdefault'

Add matomo analytics

Make columns movable

Add white space left to date/source

Make the wordspace horizontally scrollable if there are too many columns

Currently after 5 columns, the 6th get wrapped underneath so it isn't visible.

Format cards more nicely

• proper date format
• icon for source identification
• labels for kind of data
• attachment icon if applicable

Add user accounts

• login
• forget password
• save columns
• save reading position?

Allow for a user to be deleted

• deletion for fusionauth
• deletion from own tabes

Time for admin acconts?

Load columns on startup from database

Add mysql to docker compose setup

Add table to enable saving of columns

begrotingstoelichtingenen

add dialog for clicking on a document

• if there is a way to intereact with the data in jodal, show that
• else show a list of relevant tools
• add link to source

Add help text for help button

:P

Style top bar in the proper color

:)

add a wordpress for editable content

Integrate openbesluitvorming into jodal

Using the jodal_documents index

change icon for login/logout

currently not very clear

save channel upon adding it

Remove acolumn

bronnen configureerbaar maken per kolom

change title

• journalistiek dashboard lokaal
• make sure it says beta

Add login functionality

• new user registration
• login
• logout
• forgot password

integrate poliflw into jodal

using the jodal_documents ES index

bron open spending op zoekterm highlighted

Rework side pop menu to not show bogus options

:P

Multiple locations should be possible to add

Currently not becuase of a database error (which is a regression):

api-jodal_1      | 2021-05-07 12:38:06,307.307.9090118408203:app:140642026940160:ERROR:1:Exception on /columns [POST]
api-jodal_1      | Traceback (most recent call last):
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1706, in _execute_context
api-jodal_1      |     self.dialect.do_execute(
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
api-jodal_1      |     cursor.execute(statement, parameters)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 206, in execute
api-jodal_1      |     res = self._query(query)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 319, in _query
api-jodal_1      |     db.query(q)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/MySQLdb/connections.py", line 259, in query
api-jodal_1      |     _mysql.connection.query(self, query)
api-jodal_1      | MySQLdb._exceptions.OperationalError: (1241, 'Operand should contain 1 column(s)')
api-jodal_1      | 
api-jodal_1      | The above exception was the direct cause of the following exception:
api-jodal_1      | 
api-jodal_1      | Traceback (most recent call last):
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
api-jodal_1      |     rv = self.dispatch_request()
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
api-jodal_1      |     return self.view_functions[rule.endpoint](**req.view_args)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask_restful/__init__.py", line 468, in wrapper
api-jodal_1      |     resp = resource(*args, **kwargs)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask/views.py", line 89, in view
api-jodal_1      |     return self.dispatch_request(*args, **kwargs)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/flask_restful/__init__.py", line 583, in dispatch_request
api-jodal_1      |     resp = meth(*args, **kwargs)
api-jodal_1      |   File "/opt/jodal/app/resources.py", line 24, in wrapper
api-jodal_1      |     return func(*args, **kwargs)
api-jodal_1      |   File "/opt/jodal/app/resources.py", line 50, in post
api-jodal_1      |     db.session.commit()
api-jodal_1      |   File "<string>", line 2, in commit
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1428, in commit
api-jodal_1      |     self._transaction.commit(_to_root=self.future)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 826, in commit
api-jodal_1      |     self._prepare_impl()
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 805, in _prepare_impl
api-jodal_1      |     self.session.flush()
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3260, in flush
api-jodal_1      |     self._flush(objects)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3400, in _flush
api-jodal_1      |     transaction.rollback(_capture_exception=True)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
api-jodal_1      |     compat.raise_(
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
api-jodal_1      |     raise exception
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3360, in _flush
api-jodal_1      |     flush_context.execute()
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 453, in execute
api-jodal_1      |     rec.execute(self)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 627, in execute
api-jodal_1      |     util.preloaded.orm_persistence.save_obj(
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 242, in save_obj
api-jodal_1      |     _emit_insert_statements(
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1219, in _emit_insert_statements
api-jodal_1      |     result = connection._execute_20(
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1521, in _execute_20
api-jodal_1      |     return meth(self, args_10style, kwargs_10style, execution_options)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 313, in _execute_on_connection
api-jodal_1      |     return connection._execute_clauseelement(
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1390, in _execute_clauseelement
api-jodal_1      |     ret = self._execute_context(
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1749, in _execute_context
api-jodal_1      |     self._handle_dbapi_exception(
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1930, in _handle_dbapi_exception
api-jodal_1      |     util.raise_(
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
api-jodal_1      |     raise exception
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1706, in _execute_context
api-jodal_1      |     self.dialect.do_execute(
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
api-jodal_1      |     cursor.execute(statement, parameters)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 206, in execute
api-jodal_1      |     res = self._query(query)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 319, in _query
api-jodal_1      |     db.query(q)
api-jodal_1      |   File "/usr/local/lib/python3.9/site-packages/MySQLdb/connections.py", line 259, in query
api-jodal_1      |     _mysql.connection.query(self, query)
api-jodal_1      | sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1241, 'Operand should contain 1 column(s)')
api-jodal_1      | [SQL: INSERT INTO `column` (name, locations, query, user_id, `order`, src_poliflw, src_openspending, src_openbesluitvorming) VALUES (%s, %s, %s, %s, %s, %s, %s, %s)]
api-jodal_1      | [parameters: ('xyz', ['GM0518', 'GM0796'], '*', b'\xedi\x1d]?\xd6It\x98O\xfc\xadZ\x92\x17\x0f', 6, 1, 1, 1)]
api-jodal_1      | (Background on this error at: http://sqlalche.me/e/14/e3q8)

Jodal fails when adding 's Hertogenbosch and Almere

For unknown reason.... locations seems not to be updating right

Add a deployment file

Show message in column when no entries could be found

From all sources.

Add instruction to add a column if there are no columns.

Onboarding

Make using IntroJS or something similar

Need texts!

Send welcome email upon account creation

With instructions, See here

Relative time should update

Make font smaller

Like 14px instead of the 18 it is now ...

add user definable sorting per column

old -> new
or relevance

Allow for editing of query in column

Add help link

so peope can mail

Add uptime robot

We need to know if it's down

Add REST API methods for columns

Add REST API methods for columns

• create
• read
• update
• delete

Move interface to subdomain

To app.jodal.nl instead of www.jodal.nl so we can have a nice landing page in the future

Show loading thingie in column when there is no data yet

Add necessary tables to support logins

• user
• session
• ?

Add button to login in center of the screen

to make it easier