getdkan / dkan2

Deprecated: please use the 2.x branch of the dkan repo

DEPRECATED

The code for DKAN 2.x has been moved to the DKAN repository.

DKAN Open Data Portal/Catalog built on Drupal 8 - Release Candidate

As a piece of software in its Alpha cycle, the code is continuously changing and in very active development.

Features

• Harvesting of data from external catalogs that provide a data.json.
• Dataset metadata and resources.
• Web service API endpoints that allow third party applications to work with the datasets.
• Integration with a decoupled REACT front end.
• A datastore to store CSV files and make them queryable through an SQL endpoint.

License

DKAN and related modules are freely-available under the "GNU General Public License, version 2 or any later version" license.

dkan2's Issues

The datastore queue never stops

SPIKE: Determine how to build the API

We should determine what module or strategy we are using for the:

• crud APIs
• read APIs for the Interra integration

Set up CI

Headless API: Routes

See: https://raw.githubusercontent.com/interra/catalog-generate/master/build/dkan-demo/api/v1/routes.json

The schema is:

{
  "/routes.json": {
    "get": {
      "summary": "A list of routes for the catalog.",
      "description": "This list is used by the front end to determine what routes are rendered as well as the build process for rendering each HTML   page.",
      "operationId": "routes",
      "produces": ["application/json"],
      "responses": {
        "200": {
          "description": "This is a file so will either be 200 or 404",
          "schema": {
            "type": "array",
            "items": {
              "type": "object:",
              "properties": {
                "type": "string"
              }
            }   
          }   
        }   
      }   
    }   
  }
}

Set up office hours

SPIKE: Workflow for building Drupal pages w/React

In order to start building the public-facing side of DKAN in D8, and to better estimate feature tickets, we need to understand how to build react components inside Drupal pages and deploy the code.

Outcomes

• Document and create a sample of a Twig page template that contains a react component.
• Document developer workflow for editing react code and compiling/publishing

JSON form

Create a Harvest Transform that Brings Files Locally

The expected behavior should be:

1. When I run a normal working harvest (like the one in dkan_dummy_content), the dataset metadata is brought into our catalog correctly, but all the resources are urls from the orginal source.

2. We would like to create a harvest transform that will save each file from a harvest to `sites/default/file` and replace the original resource url with the newly created one.
   

3. We have some existing code that could be reused and repackaged as the transform in `dkan_harvest/src/Load/Dkan8`
   

Implementation details:

1. Create a transform that saves the resources from a dataset and replaces the original urls from the newly created ones.

2. Write a unit test that shows the transform works as expected.
   

Headless API: Search index

We want to expose the datasets like so: https://raw.githubusercontent.com/interra/catalog-generate/master/build/dkan-demo/api/v1/search-index.json

Example doc:

{
  "doc": {
    "title": "Afghanistan Election Districts",
    "keyword": [{
      "created": "2017-11-17T21:51:13.596Z",
      "title": "country-afghanistan",
      "modified": "2017-11-17T21:51:13.596Z",
      "identifier": "country-afghanistan",
      "interra": {
        "id": "country-afghanistan"
      }   
    }], 
    "publisher": {
      "@type": "org:Organization",
      "name": "Committee on International Affairs",
      "description": "The Committee on International Affairs is a working group to identify opportunities to learn from and collaborate with          government and organizations across the globe.",
      "image": "[interraUrl]/media/map.png",
      "interra": {
        "id": "committee-on-internationa",
        "source": "dkandemo"
      },  
      "identifier": "3b611ac7-17a1-48ee-a4e7-4fd08d9af3f7",
      "created": "2017-11-17",
      "modified": "2017-11-17"
    },  
    "description": "<p>This data created by the National Democratic Institute (NDI) in partnership with Development Seed, a Washington, D.C.-based    online communications consultancy, is designed to make data from the August 20, 2009, Afghanistan presidential election accessible and transparent.   We have provided this csv file as there is no download for the whole dataset. There are many more files on the website.</p>\n", 
    "modified": "2017-11-17",
    "distribution": [{
      "@type": "dcat:Distribution",
      "downloadURL": "[interraUrl]/media/district_centerpoints.csv",
      "mediaType": "text/csv",
      "format": "csv",
      "description": "<p>You can see this data plotted on a map, by clicking on 'Map' below. Individual data records can be seen by clicking on each  point.</p>\n",
      "title": "District Names",
      "interra": {
        "id": "district-names",
        "source": "dkandemo"
      },
      "schema": {
        "fields": [{
          "name": "state",
          "type": "string"
        }, {
          "name": "total.foreclosures",
          "type": "integer"
        }, {
          "name": "foreclosure.ratio",
          "type": "integer"
        }]
      },
      "identifier": "f8d6eeda-a0ca-4a0f-9bfd-7955cdbae92d",
      "created": "2017-11-17",
      "modified": "2017-11-17"
    }],
    "identifier": "c9e2d352-e24c-4051-9158-f48127aa5692",
    "interra": {
      "id": "afghanistan-election-dist",
      "source": "dkandemo"
    }
  },
  "ref": "afghanistan-election-dist"
}

Define how to name services, and implement the standard

Better Document What We Are Trying to Solve

This is to better document the specific parts of our DKAN D7 experience we would like to improve.

Some initial notes here: https://docs.google.com/spreadsheets/d/1lghBpd7DmJF-zXV4X8Fa05YH1uy2NsTQsdJh-aRzbAY/edit#gid=0

Epic: Enter metadata

User story:
As a data publisher, I can enter metadata, so that I can help consumer know what is in my data

Parent of #

Notes:

• Metadata fields will include: title, description, URL
• Dataset will have no fields
• JSON field will include title, description, and URL

Kickoff blog post

Update DKAN website

• Hide events and DKAN summit page.
• Replace California as an example
• Update features
• Add section:

DKAN in Drupal 8 is coming soon!

We’ve paid close attention to how you gather, share, and use data in the real world, and we’re keeping your favorite DKAN features while plotting out some new ones.

Meanwhile, on the back end, we’ve totally rethought how an open source open data system should be built, and DKAN2 is a Drupal 8-based, microservice-architected, schema-centered, API-first, front-end decoupled, open data platform. That means it’s more flexible, nimble, forward-thinking, and interoperable than ever before.

Stay tuned for our roadmap and keep an eye on our blog to follow the conversation as we build the next generation of open source open data tooling.

Create API submission training for non-developers

• Determine how current functionality works for non-developers

Setup Swagger page

The demo has a swagger page: http://d8.dkandemo.nuamsdev.com/api however it uses an endpoint http://admin.d8.dkandemo.nuamsdev.com/api/v1/swagger.json which hasn't been setup yet.

This is build in interra here: https://github.com/interra/catalog-generate/blob/master/internals/scripts/build.js

I would estimate 3 hours to replicate.

The import job is not being saved in the jobstore

If I run the queue, I can see that the importer is working, as data is being added to the database, but the endpoint that gives me info about the import still shows the import as null.

User story map activity

Build dataset submission UI

Headless API: Schema.json

The url for this is /schema.json.

It is somewhat defined at: http://catalog-demo.interra.io/api/

An example is here: https://raw.githubusercontent.com/interra/catalog-generate/master/build/dkan-demo/api/v1/schema.json

Here are the parts:

Collections

The collections that are included in the catalog.

Example from Interra:

{
  "collections": ["dataset", "organization", "distribution", "keyword", "license", "theme"]
}

Schema

This is the schema for the catalog. Would be great if we could just include this in a file somewhere.

Example: https://github.com/interra/catalog-generate/tree/master/schemas/pod-full/collections

It would be great if this could be a file or collection of files somewhere in dkan.

Map

This is the optional map between the schema and interra required fields. See: https://catalog-generate.readthedocs.io/en/latest/schemas/index.html#map-yml

pageSchema

This is the layout for the pages. See: https://github.com/interra/catalog-generate/blob/master/sites/dkan-demo/pageSchema.yml#L1

React JSON Schema Form integration

Setup pre-post setup of json form to handle "dkan-id"

Referenced bundles are saved in the database with a dkan-id:

• https://github.com/GetDKAN/dkan2/blob/harvest-finish/modules/custom/dkan_harvest/src/Transform/DataJsonToDkan.php#L40
• https://github.com/GetDKAN/dkan2/blob/harvest-finish/modules/custom/interra_api/src/Load.php#L101

There should be a single class to handle the above ^^^ but we can do that later.

However for the forms to work we need to remove that. This maybe could wait until #34 . Est: 2 hours.

Blog post on work in August

• JSON Schema Form
• Exploring Headless

Discovery: Full vs. Partially decoupled front end

• Determine pros and cons of each
• Estimate level of effort for progressively decoupled theme
• Determine how we handle pages and other content needs
• Make a decision

Add Breadcrumbs

Breadcrumbs don't work on the demo site because the sitemap.json endpoint doesn't include the collections. Here is what it should look like http://catalog-demo.interra.io/api/v1/sitemap.json

The one we have currently just loads the top level items:

https://github.com/GetDKAN/dkan2/blob/harvest-finish/modules/custom/interra_api/src/SiteMap.php

Estimate: 2 hours.

Audit installation documentation

Create the backlog

Docker set up

DKAN2 roadmap?

Hello,
First of all, thank you for your amazing work.
Do you happen to have a roadmap and/or timetable of the project? I'm wondering:

• at what stage DKAN2 is currently (e.g. alpha, beta, etc.)
• when you would expect the virtually production-ready DKAN2 (e.g. RC) to be ready, as well as
• whether you are planning to provide an upgrade path from DKAN.

Thanks!

Trying to import from the API runs out of memory

It seems like our docker container is set up to 128 Mb per request.

Add custom module depenencies in composer.json

We are not able to require our custom modules correctly because we aren't identifying them in composer.json correctly. See: https://www.drupal.org/docs/develop/using-composer/managing-dependencies-for-a-custom-project

I think we might need add something like:

 "../../../profiles/dkan2/modules/custom/{$name}": ["type:dkan-module"],

Determine if DKAN Tools will work on D8

Create a system to collect open api docs from subsystems

Headless API: Collections

Each content type is a collection. Each collection has a collections.json file as well as a path for all of the collections with each document: https://github.com/interra/catalog-generate/tree/master/build/dkan-demo/api/v1/collections

Add routes for resources

We might be moving away from doing this long-term but resource pages are already setup in the interra headless.

If we add resource endpoints we can have the resource previews we know and love.

To do this we just need to:

1. Change the link on the resource to include the dataset name: http://d8.dkandemo.nuamsdev.com/dataset/c9e2d352-e24c-4051-9158-f48127aa5692 (1 hr)
2. Update the doc controller to respond to the "special' case of a resource: https://github.com/GetDKAN/dkan2/blob/harvest-finish/modules/custom/interra_api/src/Controller/ApiController.php#L62

Est: 2 hours.

Interra Not Rebuilding Correctly

SPIKE: MVP API outline

In order to build DKAN 2 in an API-first way, we need a basic outline for the endpoints needed and API design we're building toward.

Outcomes

• API spec document

Acceptance criteria

• API fulfills all MVP user stories

Organization

Create an organization content type with its own schema. Use Drupal groups. Sync up groups on node save.

Relation to "React Mount Node" D8 module

"React Mount Node" D8 module provides an easy way to add React applications to a D8 site. I have tested it and it works, in a following way:

1. Define a node type that will hold react applications
2. User with special rights adds the nodes with react applications as needed. Some taxonomy terns are linked to this node type too and alow filtering of ones we need to use with specific data type or on a specific part of the site.
3. These nodes are then either embeded in another drupal node/group by users as needed (this could be potentially useful for "dkan2 data stories") or automatically shown by a view as needed.

The react app is pretty much decoupled from the rest of the drupal site. It looks for some parameters on a page where it was called (e.g. based on path or on some fields that are within the page html), and it can talk with the backend using the D8 service interface as needed, either to get additional data or to store results on the site (not really relevant but we use JSON:API). This is quite versatile and actually works quite well, but this module is stil lin "alpha" and last update was over a year ago...

If I understand it correctly, you want to achieve a similar goal, that is to embed react applications in a drupal 8 version of the dkan and use this for data visualization. How does your approach relate to "React Mount Node"?

Export Interra Components

"Index column size too large" db error during installation

I get the error message below when installing DKAN2 on Acquia. The workaround I used was to replace varchar with varchar_ascii for the fields that require 255 length in dkan_harvest.install.

Note: I do not have this issue when installing dkan2 locally via dktl.

When posting to the imports endpoint give more information

Currently when a request is malformed it returns:
{
"message": "Invalid payload."
}

Plan sprint 1

Set up Waffle

Headless API: Site Map

See: https://raw.githubusercontent.com/interra/catalog-generate/master/build/dkan-demo/api/v1/sitemap.json

This is for the breadcrumbs. Defines some routes.

Create a demo site

Update readme

• installation
• high level summary of approach (from blog post)

Define how to organize Drupal's modules and standardize

The file fetcher should remove temp files when no longer needed (upstream project work)

Create Catalog Schema

It looks like we are using a smaller schema than all of project open data. We should define the schema so we can both implement in dkan and also expose for the headless front-end.

It should be in json-schema v4 definition.