nhsuk / pomi-data-etl

Create a CSV and JSON file of POMI data for all suppliers for latest period

POMI Data ETL

ETL to download a number of data sets available under POMI collection. Patient Online Management Information (POMI) from the NHS Digital indicator portal.

The data sets used within this application are:

• Booking System
• Repeat Prescriptions
• Patient Records

Additional information is available within the indicator portal. Search for the indicator reference number i.e. P02154

Run the application

Running scripts/start will bring up a docker container hosting a web server and initiate the scrape at a scheduled time. The default is 11pm. To test locally set an environment variable ETL_SCHEDULE to a new time, i.e. export ETL_SCHEDULE='25 15 * * *' to start the processing at 3:25pm. Note: the container time is GMT and does not take account of daylight saving, you may need to subtract an hour from the time.

Further details available here.

The scheduler can be completely disabled by setting the DISABLE_SCHEDULER variable to true. This sets the run date to run once in the future on Jan 1st, 2100.

Once initiated the scrape will download the files, strip out any records that are not for the current latest period (calculated based on the records), create csv file(s) containing those records in the output dir (./html/json/) and create JSON files containing an array of objects in the form of

{
  "PeriodEnd": "dd/mm/yyyyy",
  "GPPracticeCode": "B82050",
  "Supplier": "${supplier}"
}

Where ${supplier} will be one of the suppliers listed below ["EMIS","INPS","Informatica","Microtest","NK","TPP"]. Or one of these values with an (I) appended e.g. EMIS (I). The addition of (I) represents a GP that is now using the Informatica system.

Note: The list above was created by running jq -c '[.[].Supplier] | unique ' html/json/pomi.json

Upon completion the files will also be uploaded to the Azure storage location specified in the environment variable AZURE_STORAGE_CONNECTION_STRING. Each file will be uploaded twice, once to overwrite the current file and another date-stamped file, i.e. booking.json and 20170530-booking.json.

Azure Blob Storage

If the recommended environment variables are used the JSON file created will be available in Azure storage at:

• https://nhsukpomidataetl.blob.core.windows.net/etl-output/booking.json
• https://nhsukpomidataetl.blob.core.windows.net/etl-output/scripts.json
• https://nhsukpomidataetl.blob.core.windows.net/etl-output/records.json

The Microsoft Azure Storage Explorer may be used to browse the contents of blob storage.

Environment variables

Environment variables are expected to be managed by the environment in which the application is being run. This is best practice as described by twelve-factor.

Variable	Description	Default	Required
AZURE_STORAGE_CONNECTION_STRING	Azure storage connection string		yes
AZURE_TIMEOUT_MINUTES	Maximum wait time when uploading file to Azure	10
CONTAINER_NAME	Azure storage container name	etl-output
DISABLE_SCHEDULER	set to 'true' to disable the scheduler	false
ETL_SCHEDULE	Time of day to run the upgrade. Syntax	0 23 * * * (11:00 pm)
LOG_LEVEL	log level	Depends on NODE_ENV
NODE_ENV	node environment	development

Architecture Decision Records

This repo uses Architecture Decision Records to record architectural decisions for this project. They are stored in doc/adr.

pomi-data-etl's Issues

An in-range update of nodemon is breaking the build 🚨

The devDependency nodemon was updated from 1.18.4 to 1.18.5.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

nodemon is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

Release Notes for v1.18.5

1.18.5 (2018-10-26)

Bug Fixes

• nodemon installation for node 10.x and beyond (#1441) (3afa52c)

Commits

The new version differs by 1 commits.

• 3afa52c fix: nodemon installation for node 10.x and beyond (#1441)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

An in-range update of eslint-plugin-mocha is breaking the build 🚨

The devDependency eslint-plugin-mocha was updated from 5.2.1 to 5.3.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

eslint-plugin-mocha is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

Release Notes for 5.3.0

Features

• Implement no-async-describe rule (#188)

Commits

The new version differs by 7 commits.

• d36c149 5.3.0
• 4449a3d Merge pull request #188 from papandreou/feature/no-async-describe
• 0c113b5 Simplify the fixer
• a5fa46a Fix error message
• 7c9b6dc Review fixes
• d1ffa55 Fix wrong string-less describe in test cases
• e509877 Implement no-async-describe rule

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Retry file download on network timeout

If a file fails to download due to a network timeout the download should be attempted again.

Add Snyk

An in-range update of azure-storage is breaking the build 🚨

The dependency azure-storage was updated from 2.10.1 to 2.10.2.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

azure-storage is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

Release Notes for Storage Client Library 2.10.2

2018.10 Version 2.10.2

ALL

• Upgrade xmlbuilder to 9.0.7 and extend to 3.0.2 to avoid vulnerabilities.
• Removed deprecated Buffer constructor calls in favor of static methods Buffer.from and Buffer.alloc.
• Added JSv10 link and docs.microsoft.com link.
• Improved documents.

BLOB

• Added typescript declarations to listBlobDirectoriesSegmented and listBlobDirectoriesSegmentedWithPrefix.

FILE

• Fixed an issue that empty text isn’t supported in createFileFromText.

TABLE

• Fixed an issue that uncaught TypeError could be thrown from createTable when request is not sent properly.

Commits

The new version differs by 19 commits.

• 0557d02 Merge pull request #535 from jiacfan/dev
• 47714c2 Optimize ts declarisions.
• 1600bf2 Upgrade version to 2.10.2.
• 34bf245 Update recordings for mock test.
• 1027077 Use Buffer.alloc in HmacSha256Sign.
• 23d961f Change code to use xmlbuilder ^9.0.7 and extend ^3.0.2.
• 4fd4a2f Support empty text in API createFileFromText.
• 4be9714 Upgrade extend to 3.0.2.
• e9620f7 Upgrade xmlbuilder to 9.0.7.
• 0013a55 Browser sample fixes
• 60ae42a Address uncaught TypeError in table service
• fd1e656 Remove deprecated Buffer constructor calls
• 3dd857b Fix spelling in README.md
• 327dbac Update BreakingChanges.md
• 7e16451 Fixed the heading Project Setup

There are 19 commits in total.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

An in-range update of eslint-plugin-import is breaking the build 🚨

The devDependency eslint-plugin-import was updated from 2.14.0 to 2.15.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

eslint-plugin-import is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

Commits

The new version differs by 34 commits.

• 767f01a bump to v2.15.0
• 038d668 bump webpack resolver to v0.11.0
• 20a8f3b bump utils to v2.3.0
• 1e4100d changelog note for #1261
• 548ea02 added support for argv parameter of webpack`s config-as-a-function (#1261)
• fcf2ce7 Merge pull request #1254 from ljharb/travis
• 64d9be7 allow_failures for dicey Node/ESLint intersection
• 73080d0 dep-time-travel: use older versions of dependencies
• 05c3935 repair no-deprecated for ESLint* 5
• 10c9811 node 4 fix for test util
• f165237 ah geez, bumping babel-eslint breaks no-amd/no-cjs
• acfb6e9 jk, test against eslint 2/3 but skip Typescript tests.
• b686f9d drop ESLint 2/3 from Travis/Appveyor
• cf1f6f4 [Deps] update debug, eslint-import-resolver-node, has, lodash, minimatch, resolve
• d290a87 [Dev Deps] update babylon, coveralls, eslint-import-resolver-typescript, gulp, linklocal, nyc, redux, rimraf, sinon, typescript-eslint-parser

There are 34 commits in total.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Handle networking failures when downloading

While reviewing #16 I ran the ETL process and the network dropped out which caused one of the downloads to fail (see log in PR comments).

It may be worth the code being more robust and retrying the download a (small) number of times on network failures.

An in-range update of coveralls is breaking the build 🚨

The devDependency coveralls was updated from 3.0.2 to 3.0.3.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

coveralls is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

Release Notes for Dependency security updates

As suggested by NPM and Snyk.

Commits

The new version differs by 1 commits.

• aa2519c dependency security audit fixes from npm & snyk (#210)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

An in-range update of eslint-plugin-json is breaking the build 🚨

The devDependency eslint-plugin-json was updated from 1.2.1 to 1.3.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

eslint-plugin-json is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

Commits

The new version differs by 2 commits.

• 82e0645 Replace JSHint with vscode's linting implementation
• f9f2100 Update package-lock

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Clean up intermediate files on completion

A number of csv files are left in the output folder when the ETL has completed.
These should be removed.

An in-range update of require-environment-variables is breaking the build 🚨

The dependency require-environment-variables was updated from 1.1.2 to 1.2.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

require-environment-variables is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

Commits

The new version differs by 2 commits.

• 56fe6a3 Merge pull request #7 from bjoshuanoah/issue-6
• 8c28c40 update travis requirements, update build, return object of env var values

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

An in-range update of node-schedule is breaking the build 🚨

The dependency node-schedule was updated from 1.3.0 to 1.3.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

node-schedule is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

Commits

The new version differs by 7 commits.

• c7f8827 Version 1.3.1
• 82aa564 src: avoid discarding the tz parameter
• 3ce9b06 deps: upgrade to cron-parser@^2.7.3
• 5d3777e src: make sure Invocation fireDate is a CronDate
• a6430fc fixed reschedule and plural of information
• 0754b0d doc: touch up a typo
• be9e6d8 doc: add valid values for RecurrenceRule props

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Remove use of `console.time`

Within https://github.com/nhsuk/pomi-data-etl/blob/master/app/lib/logger.js there are a couple of time* functions using console.time* to time events.

They could be removed given the logged events are time stamped.

Add bunyan logger