openreferral / hsds-transformer

Convert data into valid Human Service Data Specification (HSDS) datapackages. (Produced by the Open Referral Initiative.)

HSDS Transformer

Overview

This app allows you to convert data into an HSDS-compliant datapackage.

The Human Services Data Specification (HSDS) is a data model that describes health and human services.

This transformer currently transforms data into HSDS version 1.1.

Problem statement

Lots of people have health and human services data, but it's all in different structures and formats. Many groups want to share their data, or use data that's shared with them, but the lack of standardized data or an easy way to standardize the data presents a major barrier.

Solution

Open Referral provides a Human Services Data Specification that groups can use to share data via a common format. The HSDS Transformer tool (this project) enables a group or person to transform data into an HSDS-compliant data package, so that it can then be combined with other data or used in any number of applications.

Case Study: Illinois Legal Aid Online (ILAO)

The real-world case that's motivating this project is Illinois Legal Aid Online (ILAO), an organization that helps individuals in Illinois find legal help or resources online. ILAO has a resource directory of community services data, and they'd like to update this data to be formatted in compliance with HSDS version 1.1. They also are looking to incorporate other data sets in disparate formats and structures into their own database in order to better serve Illinois residents with more extensive and more accurate information.

This project uses ILAO's resource database in the form of three CSVs as the test case. The data was pulled in January 2019, with some records not having been updated since 2017, and therefore some of the data may not be accurate or up-to-date. The data is located in spec/fixtures/input.

Ideas for expansion

• Automate uploading to Airtable in the Open Referral Human Services Template
• Incorporate the OpenReferral Validator to automatically validate data input/output
• Plug into API-in-a-Box to automatically create a hypermedia API on top of the created datapackage
• Create a client library to generate a search UI on top of the created datapackage.

How to use this tool

First, double check whether your data is already HSDS-compliant with the OpenReferral Validator.

If it is, voila!

If it's not, proceed.

Note: Your source data must live in a single directory and be stored as CSVs.

Using this as a gem

Make sure you have bundler installed, and then run:

gem install hsds_transformer

or add hsds_transformer to your Gemfile.

Then you will be able to use the Transformer as documented below in "Transforming using the Ruby library". Instead of setting the ROOT_PATH env as the HSDS Transformer project directory, however, you will need to set this env variable as the path to the directory your source data is stored in.

Installing locally (not using as a gem)

1. Clone this repo locally.
2. In terminal, cd into the root of the hsds_transformer directory.
3. Create a new file called .env. Copy the contents of .env.example into the new .env file and update Users/your_user/dev/hsds_transformer to be the correct path to this directory on your local environment (you can run pwd in terminal to find this out).
4. Install all the gems by running bundle install

Transforming using the Ruby library

If you're familiar with Ruby and you want to use this tool in the command line, you can open an IRB shell and require the library, and begin transforming data:

1. Make sure your data is saved locally as CSVs in the transformer directory (or whatever directory you set ROOT_PATH env variable to in step 3 above).
2. Create a mapping.yaml file and save it locally in the same directory. This is what tells the transformer how to map fields from one set of CSVs into the HSDS format. See spec/fixtures/mapping.yaml for an example.
3. Open up an interactive Ruby session in terminal by running irb (or pry - up to you!)
4. Require the class: require "./lib/hsds_transformer"
5. Run the transformer:

HsdsTransformer::Runner.run(input_dir: "/path/to/input/", mapping: "/path/to/mapping.yaml", output_dir: "/path/to/output/")

6. Now check the tmp directory for your newly created HSDS files!

You can also pass two additional arguments to the .run command: include_custom and zip_output. The output is by default not zipped, but if you want it to be, you can pass true as the value of this argument:

HsdsTransformer::Runner.run(input_dir: "/path/to/input/", mapping: "/path/to/mapping.yaml", output_dir: "/path/to/output/", zip_output: true)

If your input data includes non-HSDS fields you want to see in the output files as well, you can pass true for include_custom.

HsdsTransformer::Runner.run(input_dir: "/path/to/input/", mapping: "/path/to/mapping.yaml", output_dir: "/path/to/output/", include_custom: true)

Using the API without Docker

If you don't want to use this as a Ruby library, you can use it as an HTTP API. Start the API from the root of the project directory:

rackup -p 4567

Make a POST request with params: input_path, mapping, include_custom. Each of thse should contain a path to a CSV file and a mapping file (for mapping).

E.g.

curl -X POST -F "input_path=/Users/gwalchmai/Dev/hsds_transformer/spec/fixtures/input" -F "mapping=/Users/gwalchmai/Dev/hsds_transformer/spec/fixtures/mapping.yaml" http://localhost:4567/transform

The response will be a zip file of the transformed data. You can also pass add -F "include_custom=true" if your input data has custom non-HSDS columns you wish to include.

The API then streams a zip file back with the properly transformed data. The zip is also saved locally on the API server (maybe your local env) at data.zip in the root directory

Using the API with Docker

Before anything else, make sure you have docker installed and running.

First, build the image locally:

docker build . -t transformer_api

Now, run it:

docker run -p 4567:4567 transformer_api

You should now be able to interact with the API at http://localhost:4567.

Custom Transformers

The BaseTransformer maps data from the input directory to compliant HSDS datapackage and CSVs using the mapping.yaml, and it requires a pretty one-to-one and straightforward mapping. You may need additional cleanup, parsing, or mapping, such as parsing out schedule text. If so, you can create a custom transformer and specify it when running the script or using the API. Check out the lib/hsds_transformer/custom directory for examples.

1. Write your custom transformation code.
2. Save it as a class in lib/hsds_transformer/custom following the naming conventions already there.
3. Add the class name to the array of valid custom transformers in the HsdsTransformer::Runner class.
4. Specify this custom transformer when invoking the transformer:

HsdsTransformer::Runner.run(custom_transformer: "Open211MiamiTransformer", input_dir: "/path/to/input/", mapping: "/path/to/mapping.yaml", output_dir: "/path/to/output/")

or when making a request to the API:

curl -X POST -F "custom_transformer=Open211MiamiTransformer" -F "input_path=/Users/gwalchmai/Dev/hsds_transformer/spec/fixtures/input" -F "mapping=/Users/gwalchmai/Dev/hsds_transformer/spec/fixtures/mapping.yaml" http://localhost:4567/transform

Examples

You can find examples of data and mappings in the examples directory.

Related Projects

Open Referral Playground App

• Playground with various tools and code for working with Open Referral data
• Github

Open Referral Drupal

• Info
• Github

Open Referral Wordpress

• Github

Open Referral Laravel Services

• Laravel/MySQL/Vue.js app
• Github

Open Referral Validator

• Docs
• Github

Open Referral datapackage code

• Repo containing some code exploring creating Open Referral Datapackages
• Github

Open Referral Sample Data

• Repo of sample data for use in projects
• Github

Open Referral Gem

• This doesn't seem to have any actual code yet, but maybe the maintainer will update!
• Github

Open Referral OpenAPI Specification

• Docs
• Githbu

hsds-transformer's Issues

Add HTTP API to expose services

This should be a POST request that kicks off the OpenReferralTransformer class to process the provided files. The request can either be a multipart form request that streams that file contents to the API, or the request could reference the web URIs where the files live. If the latter, then the API needs to pass those URIs to the OpenReferralTransformer which should be able to get them during the transform process.

Turn into a gem

It would be useful to have this packaged and available as a gem so that it can be used in other projects as a dependency rather than solely as a microservice API or command line tool.

Other sample data to test

We recently had a set of sample data that came without a datapackage (and, also, I think some of the relationships between fields may not be carefully defined). So you can test with some of Florida's legal aid providers: https://drive.google.com/drive/folders/1UghRs1TnyrhgoVj-s6P0XqtA0Ic8erEl

(FYI, please only use this data for a transformation test, it's already out of date and incomplete, and wasn't intended for like publishing on the internet.)

Automate validation with OpenReferralValidator

Would be super useful to validate incoming data and transformed datapackages against https://github.com/spilio/openreferral-validator

Add Miami custom transformer tests

Need to add unit tests for the custom transformation code in the Open211MiamiTransformer class. 🙀

Save phone numbers as individual phone records

Right now there are phone numbers present in the locations, organizations, and services data. In HSDS, phone numbers are stored in their own table and linked to their parent record via a foreign key (e.g. service_id).

We want to pull out the phone numbers from these 3 input files and save them as phone records in a new phones.csv and maintain the correct foreign key links.

https://openreferral.readthedocs.io/en/latest/hsds/reference/#phone

Save addresses from Org and Locations tables as Address records

An address may be part of an incoming organization row or location row. We should save this as its own row in an addresses.csv with the correct foreign key linking it to the original organization or location.

Create project page with docs

Github pages site that gives overview of the tool, documentation, and relevant links. See this page for the Open Referral Validator for inspiration.

Smart phone number creation

When we collect phone data from organizations, services, and locations, we're taking the whole field and putting it in the number column in the output phones.csv. This is a problem because the phone field might actually include extensions or other text. It would be great to at least parse out extensions in the number column and move them to the extension column.

See examples in this fixture: https://github.com/switzersc/open_referral_transformer/blob/master/spec/fixtures/output/resources/phones.csv

What are the basic data structure constraints for this to work?

Does the input data need to have some basic structure in order for us to map? Should there be at least an organizations and locations file?

Airtable link

Hi, it seems the airtable link https://airtable.com/universe/expTMdQFD5r9G6V9Y/open-referral-human-services-template doesn't resolve and brings up a 404 for me? I have tried this while logged in as well.
For context, I am looking at how we align the Leeds LOOP (repository) to openreferral, and adopt it with contributions to support the city's services, information and local 'language'

Use datapackage-rb gem to infer and generate datapackage.json from transformed data

Right now we do not dynamically generate the datapackage.json file based on the contents of the transformed data directory, but we should. This is because transformed data may not include all HSDS models or fields, and may include additional custom fields, so using a static datapackage.json file would not be capture these idiosyncrasies. We want to use the datapackage-rb gem to do this, but this feature is not supported yet so is dependent on a PR to that gem to add the functionality. Once that's done, we can utilize the gem here to create the JSON file.

Naming convention: hsds-transformer?

The other repos have a dash rather than an underscore in their name. Can i change this to hsds-transformer without breaking anything?

Add eligibilities.csv output

We need to generate eligibilities from input data. This is usually one text field in a CSV.

Headers: id, service_id, eligibility

See the HSDS reference: https://openreferral.readthedocs.io/en/latest/hsds/reference/#eligibility

For the test data mapping, you can add this to the locations segment:

Income standard for eligibility:
  model: eligibilities
  field: eligibility
  foreign_key_name: service_id
  foreign_key_value: Services

Failing spec: It "converts a services file into valid HSDS services data"

This spec is failing because the services csv in spec/fixtures/input has an invalid CSV line at the end. We need to add some layer of validation during transformation that removes invalid lines.

Save regular schedule in HSDS format

This will likely involve some regex parsing for dates and times in a string column.

https://openreferral.readthedocs.io/en/latest/hsds/reference/#regular_schedule

Make license

license this boi

Fix the build

Build is failing on travis CI (though not locally), due to file reference issues. Should move to using the Fake File System gem for tests anyway, which should resolve the problem.

Add ability to specific output path

Right now everything gets piped to tmp after being transformed, including the results of the tests. This isn't great because files will get overwritten when tests or new transformations are run. We should give the user the ability to tell the transformer to what path their files should be sent.

API Not Working (Using the API with Docker)

Using the API with Docker

Hosted the project on Docker & trying API "/transform" method POST from the postman, Getting an error "A mapping file is required".

I have added the
input_path : /spec/fixtures/base_transformer/input
mapping : /spec/fixtures/base_transformer/mapping.yaml

But still getting the error "A mapping file is required".

Add service_at_locations csv

We need to create the service_at_locations CSV in HSDS format: https://openreferral.readthedocs.io/en/latest/hsds/reference/#service_at_location

You will need to add this to the locations section in the mapping.yaml file:

Services:
  model: service_at_locations
  field: service_id
  foreign_key_name: location_id
  foreign_key_value: Location_id

Dockerize

Add a Dockerfile with the current ruby version and build steps (bundle install primarily).

Add ability to append multiple input fields to one output column

We want to use multiple input fields in one output field. For example, in the services model in HSDS, there is an application_process text field. In our fixture input data, the services.csv contains multiple fields that are relevant for the application process: Application form link, Process for applying by email, Process for applying by phone``Process for applying online, Process for walk-ins. We should have a way to append all of these fields into a single output column.

This is what I'm thinking for an addition to services in mapping.yaml:

Process for applying by email
  model: services
  field: application_process
  append: true
Process for applying by phone
  model: services
  field: application_process
  append: true
Process for applying online
  model: services
  field: application_process
  append: true
Process for walk-ins
  model: services
  field: application_process
  append: true

and the append: true field indicates that these should be appended to the output field, so that all the above are added into the same text string in application_process.