getdkan / catalog-generate

Interra Data Open Data Catalog generator. Demo: http://catalog-demo.interra.io/

Interra Catalog Generate

Open Data should be easier.

This project generates an Open Data catalog as a static html site. It can be hosted on Github, S3, or any generic server. The schema is declared using JSON Schema in Yaml files. The metadata and all content are stored in JSON files.

Features

• CKAN / DKAN style data catalog
• Full-text search using Elasticlunr
• Easy to update and override schema
• Simple data workflows
• Harvesting from outside catalogs and data sources
• Easy-to-update or override React front end
• Full administrative interfeace using Interra Catalog Admin
• Low-cost hosting using S3, github pages, or other static file server
• Additional features as microservices

Reduced Overhead

Traditional catalogs use a SQL database (MySQL, PostgreSQL), webserver (Nginx, Apache), and search engine (SOLR, ElasticSearch) to provide a simple catalog. Many catalogs consist of metadata for less than 1,000 datasets with some additional features such as a datastore or visualization library. Interra drastically reduces the overhead for publishing data, especially for catalogs with under 1,000 records, because a database, webserver (if published to S3), and search engine are not necessary. Interra uses Elasticlunr as a search index which offers some features search engines such as boosts and query tokens without the overhead of that service.

Storage

Storing data as static files, or optionally in a NoSQL store like MongoDB, drastically reduces the overhead for managing and publishing data. Catalogs using SQL databases rely on complex abstractions for translating data from open formats to an internal state and then back to open formats. For example, many data workflows involve ingesting JSON files, writing them to rows and tables in a SQL database using an ORM, changing the data, then exporting them back to JSON files. These catalogs also require a web server and scripting language to receive requests and serve JSON.

Interra Catalog takes a different approach. Data is saved in a document store which is by default a file system. The only change in the way the data is stored versus consumed are internal references between docs. Updating data between harvesting, storing, and publishing is done with a small set of javascript functions. There is little state to understand and maintain.

Publishing

Documents are exported as JSON files and rendered using a React app. Search is provided using Elasticlunr.

Additional Features

Additional catalog features are added as microservices. Interra Catalog Admin offers a user-interface for creating, editing, and updating data. A datastore is coming soon using CARTO. Data visualization dashboards will be added with React Dash.

Extending

The schema is extremely flexible and easy to update. Adding a new field is as simple as adding lines to a configuration file. Example schema's include Project Open Data's schema however any schema that has the concept of a "primary collection" such as a "dataset". See the test suite for an example.

Roadmap

Roadmap soon to be shared. This project is still in early development.

Example Site

Currently have an early POC here: http://healthdata-example.interra.io.

Getting Started

To get started install node modules:

npm install

The default search uses ElasticLunr. We require the latest 0.9.6 version however the current version on npm is 0.9.5. To get the latest version:

rm -r node_modules/elasticlunr; git clone [email protected]:weixsong/elasticlunr.js.git node_modules/elasticlunr;

To create a new site start the plop generator:

node_modules/.bin/plop

Documentation

Documentation can be found at Read the docs.

Requirements

Requires node v6+. This is built to run using Linux. Windows support will not be provided.