openreferral / deprecated-hsds-1.1-validator

Validator micro-service for Open Referral data resources

Open Referral - Validator microservice

Visit Project Github page.

A micro-service with a RESTful API for validating data packages and resources using the official Human Services Data Specification from Open Referral. Thanks to @spilio for development of this tool.

Running the service

Running as a Docker container

You need to have Docker service installed on your local machine or any other target host.

Pulling the image from Docker Hub

The fastest way to get started is to pull the latest pre-built image from the Docker Hub registry by running:

$ docker pull openreferral/validator

Building the image locally

Otherwise you can build the image on your local machine by running the following command within the project's directory:

$ docker build --tag "openreferral/validator:latest" .

Running the service container

After the Docker image is available you can launch a container by running:

$ docker run -d --network=host --name=openreferral-validator openreferral/validator:latest

You can use any name you want, by replacing the "openreferral-validator" value with one of your choice. The container will bind to localhost:1400 by default. You can change the host and port of the container by setting the HOST and PORT environment variables like so:

$ docker run -d --network=host -e HOST=localhost -e PORT=1300 --name=openreferral-validator openreferral/validator:latest

Once the container is launched, you can stop it and start / restart it on demand like so

$ docker stop openreferral-validator
$ docker restart openreferral-validator

Running locally with NodeJS

In order to run the project locally you need to have the latest NodeJS and npm installed.

Once your environment is all set up, clone the repository, go into the root directory and install all dependencies by running

$ npm install

Once all the dependencies have been downloaded, run the application with

$ npm start

Using the validator service

Once the service has been launched you can verify that the API is up by hitting http://localhost:1400/health. If everything is ok you should get a blank page (or a 200 response).

OpenAPI definition

The micro-service has an OpenAPI 2.x compliant definition that is automatically generated on startup. You can find the OpenAPI (Swagger) definition here http://localhost:1400/swagger. You can parse the OpenAPI definition with any popular API tool like Swagger, Postman, etc and start interacting with the service.

API operations

GET /health

Description

Returns a 200 OK response

GET /validate/datapackage

Description

Validate an HSDS data package. The operation requires the URI of valid datapackage.json file to be provided as a query parameter. The service will parse the contents of the data package and try to validate all enlisted resources.

Query parameters

• uri: A valid local or remote URI of a datapackage.json descriptor file.
• relations: (optional) A boolean flag indicating whether to enable data relations check through defined foreign keys - default is false

Response

The operation returns a collection of validation results, one per resource as defined within the data package descriptor.

Example call

Given we have a sample datapackage.json file at http://example.com/openreferral/datapackage.json that contains a small subset of resources, we can run the following command to validate the contained resources:

$ curl 'http://localhost:1400/validate/datapackage?uri=http://example.com/openreferral/datapackage.json'

If all data resources are valid, the service will return a response like:

[{
  "valid": true,
  "resource": "organization"
}, {
  "valid": true,
  "resource": "program"
}]

In case something is wrong, the response would be something like:

[
    {
        "valid": true,
        "resource": "organization"
    },
    {
        "valid": false,
        "errors": [
            {
                "row": 3,
                "description": "Foreign key \"organization_id\" violation in row 3"
            }
        ],
        "resource": "program"
    }
]

POST /validate/csv

Description

Validate an HSDS data resource file. The operation validates an uploaded CSV data stream using the definition of a specified resource as found in the standard Open Referral data packagespecification. Clients should send a form payload containg a type field with the name of the HSDS logical resource and a file that contains the CSV data stream.

Consumes

• multipart/form-data

Payload

• type: A valid HSDS resource name, e.g. contact
• file: The CSV file to be validated

Example call

$ curl -F 'type=contact' -F 'file=@/home/chris/contacts.csv' 'http://localhost:1400/validate/csv'

A successful validation would return something like:

{
    "valid": true,
    "errors": []
}

Sample data sets

You can test drive the validator using the HSDS sample data sets found in the dedicated Github repository. Clone the repository or download as a ZIP archive and extract them locally.

deprecated-hsds-1.1-validator's Issues

sample-data datapackage hosted on GitHub doesn't validate

http://localhost:1400/validate/datapackage?uri=https://raw.githubusercontent.com/openreferral/sample-data/master/simple/datapackage.json

The connection was reset

If I host the package myself, it validates fine. I suspect this has something to do with GitHub's headers.

Error log:

$ docker logs openreferral-validator

> [email protected] start /app
> node src/app.js

info: Server launched [0.0.0.0:8004]
/app/node_modules/csv-parse/lib/index.js:369
      })(this))) || (l - i <= this.options.delimiter.length && this.options.delimiter.substr(0, l - i) === remainingBuffer) || (l - i <= this.options.escape.length && this.options.escape.substr(0, l - i) === remainingBuffer)) {
                                                     ^

TypeError: Cannot read property 'length' of null
    at Parser.__write (/app/node_modules/csv-parse/lib/index.js:369:54)
    at Parser._transform (/app/node_modules/csv-parse/lib/index.js:207:14)
    at Parser.Transform._read (_stream_transform.js:186:10)
    at Parser.Transform._write (_stream_transform.js:174:12)
    at doWrite (_stream_writable.js:396:12)
    at writeOrBuffer (_stream_writable.js:382:5)
    at Parser.Writable.write (_stream_writable.js:290:11)
    at IncomingMessage.ondata (_stream_readable.js:639:20)
    at emitOne (events.js:121:20)
    at IncomingMessage.emit (events.js:211:7)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `node src/app.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-07-15T15_39_36_668Z-debug.log

Change name to hsds-validator

@spilio would it be hard to change the name to hsds_validator?

400 is returned for all requests that don't pass validation

Right now the validator API returns a 400 Bad Request whenever a request is made with a file that doesn't meet validation requirements. This was pretty confusing because 400s are meant to be returned for malformed syntax, not semantic issues or otherwise syntactically valid requests. This error code led me to believe there was something malformed about my file or HTTP request, and it took about an hour of debugging and head-scratching for us to realize that the request and the CSV themselves were well-formed but that our CSV just didn't pass the validation logic.

Notably, when using curl you probably won't see the 400 status code response unless you're using the verbose flag, and you will see the validation message; however, when using Postman or in our case Ruby HTTP clients, only the error is surfaced, not the validation message in the body.

I'd advocate for updating the validation API to return a different error status code (e.g. 422 - semantically bad request), or to return a 200 with the validation message in the body. I'm suggesting a 200 because the API request itself is actually valid: the endpoint is performing the function we expect, which is checking the received entity against some logic, and the response includes information that we've asked for (the validation message). Therefore as a client, everything is OK, I've just now gotten info that I have work to do.

Thoughts?