afoone / opencrvs-core

A global solution to civil registration

OpenCRVS

A global solution to civil registration
Report an issue · Case studies · Implementations · About us

• OpenCRVS
• Read the docs
• Preparing for installation
• What are the key OpenSource dependencies of OpenCRVS?
• What is inside the OpenCRVS packages?
• What about end-to-end UI testing?
• Why Docker Swarm? ...and is there Kubernetes support?
• OK. How do I set up a development environment?
• How do I configure and localise OpenCRVS?
• How can I test OpenCRVS locally on an Android device?
• How can I set up continuous integration and delivery?
• How can I install and manage an OpenCRVS server cluster?
• Clearing and creating your own reference data in your resources module
• Create a new metadata db dump in your resources module
• Docker scripts
• How can I deploy to a staging environment cluster?
• How can I deploy to a QA environment cluster?
• How can I deploy to production?
• How do I export recent registrations?
• How does OpenCRVS back up registration data?
• Become part of the OpenCRVS Community
• Contributing
• How to tag an OpenCRVS release?
• Is OpenCRVS live anywhere, and what functionality is currently available in the latest release?
• License

OpenCRVS

We are on a mission to ensure that every individual on the planet is recognised, protected and provided for from birth.

OpenCRVS (Civil registration & Vital Statistics) is a digital public good to help achieve universal civil registration and evidence-based decision making in all country contexts.

It is a full stack & fully configurable solution for the interoperable, civil registration services of a country. Built with microservices, health service compliant HL7 - FHIR standard NoSQL databases, containerised and distributed on Docker Swarm for private or public cloud infrastructure, and accessible to front line staff by a multi-lingual, offline & low connectivity capable mobile web application, an admin system and a performance analytics dashboard.

Read the docs

Before going further, you should read the Documentation.

A companion example configuration; the OpenCRVS Resources package for Zambia, can be found on GitHub.

Both OpenCRVS Core and a Resources package are required to run OpenCRVS. When you configure OpenCRVS for your country, you will need to duplicate and make your own Resources package.

Preparing for installation

• Read the technical documentation.
• OpenCRVS uses some exciting tools. Take some time to understand the dependencies below.

When you clone this OpenCRVS Core repository, you are downloading:

• All of our microservices and the mobile, progressive web application as Lerna packages.
• Docker compose files to run a development environment and deploy OpenCRVS.
• An Infrastructure folder, containing Ansible setup, OpenHIM configuration, Netdata configuration, Hearth plugins, ElasticSearch configuration, Traefik configuration and a number of essential scripts required to automate deployment, netweorking and backup.

Now, take some time to understand the directory structure. Read more about what all the packages are for below.

Now you are ready to follow the step-by-step development environment installation instructions. At a high level you will be required to:

• Install some basic dependencies for your workstation.
• Clone the repo and install some application dependencies.
• Run a single command to start OpenCRVS Core and the OpenCRVS Resources module in Docker on your workstation
• Install a base configuration for OpenHIM
• Populate your local Mongo DB with reference data and test users for the Zambia configuration.

If you have any issues, we would love to know so that we can improve our documentation. Please talk to us on Gitter.

What are the key OpenSource dependencies of OpenCRVS?

When you run an installation, the following dependencies are automatically provisioned alongside the OpenCRVS Core on your infrastructure in docker containers.

Hearth MongoDB Database layer

In order to support configuration for limitless country scale, OpenCRVS was designed for NoSQL, built on MongoDB, and aligned to a globally recognised healthcare standard.

Massively scalable and extensible, Hearth is an OpenSource NoSQL database server built by the OpenCRVS founding member Jembi Health Systems, using interoperable Health Level 7 FHIR v4 (ANSI Accredited, Fast Healthcare Interoperability Resources) as standard.

We innovatively extended FHIR to support the civil registration context. Our civil registration FHIR standard can be contributed to in this repository at Jembi.

ElasticSearch

De-duplication management to ensure data integrity is essential to any respectable civil registration system. A fast search engine lowers operational costs and improves the user experience for frontline staff.

OpenCRVS uses ElasticSearch, an industry standard, NoSQL document orientated, real-time de-duplication & search engine. Lightning fast, intelligent civil registration record returns are possible, even with imprecise, “fuzzy” search parameters.

InfluxData

Hyper efficient and optimised, Influx is a time series database for big data insights. Millisecond level query times facilitate civil registration statistical queries over months of data, disaggregated by gender, location and configurable operational and statistical parameters.

OpenHIM enterprise service bus, interoperability Layer

The OpenHIM (Health Information Mediator) is a NodeJS enterprise service bus designed to ease interoperability between OpenCRVS and external systems such as Health & National ID. It provides external access to the system via secure APIs. OpenHIM channels and governs internal transactions, routing, orchestrating and translating requests into FHIR between services and the database layer.

What is inside the OpenCRVS packages?

The core of OpenCRVS is a monorepo organised using Lerna. Each package reorts unit test coverage in Jest. Following the microservice, 1 service per container model, every package is independently scalable in a single docker container.

OpenCRVS microservice business layer packages

The OpenCRVS’ back end microservice architecture enables continuous evolution of its business requirements.

The microservices are written in TypeScript (a strictly typed superset of JavaScript that compiles to JavaScript) and NodeJS using the fully documented HapiJS framework.

Each microservice in OpenCRVS has no knowledge of other services or business requirements in the application, and each exposes it’s capabilities via JWT secured APIs.

Microservices:

1. auth - the authentication microservice for OpenCRVS, JWT token generation and management in Redis.

⋅⋅⋅Our client applications are protected by SMS 2-Factor Authentication. Our apps and microservices utilise OAuth best practices for JWT tokens.

2. commons - a shared library package for Node methods used by multiple microservices.

3. gateway - the GraphQL and Apollo API gateway for the OpenCRVS client.

Using GraphQL allows OpenCRVS to perform much faster and more responsively in remote areas by drastically reducing the number of HTTP requests that are required to render a view in the presentation layer.⋅⋅

The OpenCRVS GraphQL Gateway is a JWT protected Apollo server that requests and resolves FHIR resources from Hearth via OpenHIM into GraphQL, for easy consumption in the client applications.

• metrics - the civil registration metrics and analytics microservice using the Influx time series database.

• notification - the microservice that manages SMS communications from OpenCRVS, including content management and SMS supplier details.

• search - the search microservice for OpenCRVS using ElasticSearch

• user-mgnt - the user management microservice for the OpenCRVS client. User permissions and roles can be centrally managed, supporting IT organisations that conform to ISO27001 certification.

• workflow - the OpenCRVS business process orchestration microservice, mediating civil registration vital event status and audit updates.

OpenCRVS client application packages

• login - the login UI client built in React.

• client - the OpenCRVS UI client for civil registration built in React.

Client npm dependencies and enablers include:

• Easy build configuration with create-react-app, craco, typrescript-eslint

• Multi-lingual content management support using react-intl

• ES6 JS component styling using styled-components

• Fully configurable, high performance form management using formik

• Pure JavaScript, client side, offline PDF certificate generation using pdfmake

• Read-only application state management using redux

• Unit tests coverage with Jest & Enzyme UI component tests.

Using an Android progressive web application for our client app means that we can take advantage of offline functionality and native mobile features using Workbox, without the TCO overhead of maintaining multiple web and mobile codebases and respective App/Play Store releases.

In remote areas, registrars can save a configurable number of registrations offline on their mobile phone using IndexedDB.

OpenCRVS component library package

components - a UI component library package for the clients using React Styleguidist.

OpenCRVS performance testing packages

• integration - performance tests for OpenCRVS using the K6 framework.

What about end-to-end UI testing?

Besides the thorough unit testing coverage with Jest, we supply e2e UI test scripts using Cypress. Built in modern JavaScript with Typescript support, Cypress beats Selenium's capabilities in multiple ways.

Because the OpenCRVS UI is completely configurable to your country, the end-to-end testing scripts are located in the companion reources module described in the development environment setup instructions in this README. The resources module is bespoke for your country and we release a Zambian example.

Why Docker Swarm? ...and is there Kubernetes support?

Docker Swarm was chosen for it's simplicity, so that previously unskilled system administrators can quickly up-skill in the techniques of private and public cloud infrastructure management. We wanted to democratise the containerisation benefits of AWS/Kubernetes style public cloud deployments for developing nations.

Some nations may be located far from a developed world datacentre. Many nations may not be able to legally support international data storage of citizen data. Often getting the legal approval requires regulatory change which obviously can take some time. In the short term, these nations may not have access to the development skills necessary to manage a complicated distributed cloud deployment, so ease-of-use is paramount.

Docker Swarm makes it easy to commence service distribution privately and then migrate publically when an organisation is ready to do so. Docker Swarm automatically configures a "round robin" load balanced cluster, and provides Service Discovery out-the-box.

We are working on a Kubernetes Software-As-A-Service solution, so that smaller nations can hand over system administration to a 3rd party to manage solely in the public cloud, if these nations can get regulatory approval.

OK. How do I set up a development environment?

Now that you have a good backround in the OpenCRVS core, it is time to set up a local development environment.

1. First, make sure your environment is prepared by installing these pre-requisites:

   • Node.js - using node version manager is also useful for installing node.
   • Yarn
   • Docker - if on linux you will need to make sure docker can be run by your user, not only by root or using sudo - see here.

   On Linux you will also need to:

   • increase your file watch limit using: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
   • increase vm max heap for ElasticSearch using: echo vm.max_map_count=262144 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

   On Mac you will need:

   • Docker For Mac

2. Next, clone this repo! :) git clone https://github.com/opencrvs/opencrvs-core.git

3. cd into the repo and run chmod 775 data/elasticsearch from root of the project to allow ElasticSearch access

4. Run yarn to install deps

5. Run docker swarm init - your localhost has to be a Docker swarm manager in order to use the "overlay" network.

6. Next, OpenCRVS depends upon your own country specific "resources" module of reference data (addresses, employees, offices, facilities), custom configuration of vital event registration forms and your own integrations.

A test "resources" module for Zambia is released for free.

cd ../ back up and clone this example Zambia resources module. You can iterate upon it for your needs.

git clone https://github.com/opencrvs/opencrvs-zambia.git

7. Set the following environment variables locally. The first is the path locally to where you just cloned the resources module:

export RESOURCES_PATH=<path to your resources module>

The second is the country code you are using in the resources module, eg: zmb

export COUNTRY_CODE=zmb

You need to pass a comma delimited string of supported languages to the yarn dev command in order to start the stack.

OpenCRVS has the capability to be fully multi-lingual and languages can be configured.

Client Application: Internationalisation and languages can be configured in client.json.

SMS Notifications: Internationalisation and languages can be configured in notification.json.

We have provided some handy tools in the test "resources" module for Zambia to help you load your languages into a content management system such as Contentful.

OpenCRVS currently supports the standard Roman and Latin character set and Bengali. In OpenCRVS Alpha, we will need to assist you to configure core to support a new language in the language select in a pull request. We will gladly provide support to you if you want to provide translations and hugely welcome all localisation efforts.

Currently we only have Bengali and English translations. If you are planning to introduce a new language, please talk to us on Gitter and submit a new translation file. We can easily enable the new language string in a PR.

Currently the supported language strings can be set to either:

Bengali and English

"bn,en"

English only

"en"

In the Zambia resources package you can see how to set up OpenCRVS with a Content Management System.

Run yarn dev <supported-languages> to up the OpenCRVS core dev environment (frontend and backend services in this repo start as local dev servers that will autoreload and dependencies are started via docker-compose) OR alternatively you may run the dependencies and the services in this repo separated in two diffrent terminal with yarn compose:deps (dependencies) and yarn start manually setting the supported languages string in a .env file. We also provide tmux commands.

If you are using OSX:

Docker For Mac can affect OpenCRVS ability to find containers on localhost. Find your local IP address and start the dev environment, replacing the IP address with yours, like this: LOCAL_IP=192.168.0.5 yarn dev <supported-languages>

8. cd opencrvs-zambia into the resources repo you previously installed and run yarn to install deps

9. Run CERT_PUBLIC_KEY_PATH=<where your core repo is>/.secrets/public-key.pem yarn start to run the resources module

Once your Zambia resources module is running, you will now need to populate your database with test data using the Zambia implementation.

10. Populate the database with test data from your resources module

Run yarn db:backup:restore to populate the database with reference data and test users for Zambia.

That's it! You should be running OpenCRVS with test users and test locations. Apps can be found running at the following URLs:

• Login: http://localhost:3020/
• Client: http://localhost:3000/
• Styleguide: http://localhost:6060/

Login using using the following Zambia user accounts.

• Field agent: kalusha.bwalya
• Registration Agent: felix.katongo
• Registrar: kennedy.mweene
• System Administrator: emmanuel.mayuka

In development, the password is "test" and the 2-factor auth SMS code is "000000".

Please get in touch with us to find out more about the available business functionality for each user type, or talk to us on Gitter.

After logging in you should be redirected to: Client: http://localhost:3000/

UI component library will be running here: Styleguide: http://localhost:6060/

How do I configure and localise OpenCRVS?

OpenCRVS depends upon your own country specific "resources" module of reference data (addresses, employees, offices, facilities), custom configuration of vital event registration forms and your own bespoke API integrations.

A test "resources" module for Zambia is released for free in order to illustrate how this can be done.

Just follow the instructions in the Zambia resources package or please get in touch with us for help. We'd love to hear from you!

How can I test OpenCRVS locally on an Android device?

You can run a PWA on an Android device and connect to OpenCRVS running on your local network. This is the best way to work and debug service workers and do actual device testing.

1. Register to https://ngrok.com/. The free plan is enough for this.
2. Go to https://dashboard.ngrok.com/auth and get yourself an auth token
3. Create a .env file to packages/mobile-proxy

AUTH_TOKEN=THE_AUTH_TOKEN_YOU_GOT_HERE

4. Start the development environment as you normally would, but add --ignore @opencrvs/client to the start command. E.G. yarn start --ignore @opencrvs/client

5. Run yarn start in the mobile-proxy package

6. You should now have an ngrok url that can be used remotely on any device. It's still required to be in the same network as the host machine, as some services (login) aren't used through ngrok.

7. Build the client app NODE_ENV=production yarn build

8. Serve the client up in 3000 port by running yarn serve

9. You can now open up the address you got from the mobile proxy in your mobile browser

How can I set up continuous integration and delivery?

We provide an example Travis configuration to automate unit and end-to-end testing integration & deployment.

How can I install and manage an OpenCRVS server cluster?

OpenCRVS should be deployed on a minimum cluster of 3 nodes, each with the following minimm specification:

8 GB Memory (preferrably 16 GB) / 160 GB Disk / Ubuntu 18.04.3 (LTS) x64

To prepare your server cluster and manage the Docker Swarm, some pre-requisites and instructions are documented here

An Ansible playbook script is provided here to automate the vast majority of your server cluster setup.

Clearing and creating your own reference data in your resources module

You must read the full instructions in the companion resources module to create your own reference data. The following commands help you get back to an empty state.

1. cd opencrvs-zambia to return to the Zambia resources module

2. Run yarn db:clear:all to delete the entire local database

3. Log into the OpenHIM at here to load the initial base config - default password is [email protected]:openhim-password (login will fail a security check as we are using self signed certs by default, follow the instructions in the error message so that Chrome recognises the SSL cert.)

4. Once logged in click Export/Import then drop the file infrastructure/openhim-base-config.json into the import box and click 'Import'

5. Click Channels and check all have loaded successfully.

6. Click edit, and then go to routes tab and change the value of host from service name to your local IP address.

7. Test the setup with curl http://localhost:5001/fhir/Patient/123 you should get some JSON with a 'Not found' error.

Create a new metadata db dump in your resources module

Start the development environment as described above, then:

1. Start the dev environment - as explained above.

2. Code reference data for your country requirements following instructions inside the README of your resources module. Then populate the database like this: yarn populate:<<insert alpha3 country code>> && cd ../..

3. Create new backup zip files to act as your baseline. yarn db:backup:create <<insert country code>>

4. Commit and push the new db dump archive files that have been created in your country folder in resources package to your private repo.

Docker scripts

There are a number of docker scripts available via yarn. If you need to manage the docker containers, some of these scripts may be useful.

The yarn compose:* scripts only setup the dependencies in docker containers and not the applications in this repository.

• *:build scripts which just build the images

• *:up scripts which just run pre-build images in containers

• *:down scripts which stop and remove the containers (along with data not stored in a volume!)

A number of other useful Docker commands you will need in order to manage the swarm cluster are accessible here

How can I deploy to a staging environment cluster?

To deploy to a staging environment we combine docker-compose files that are used in the docker setup above with a few others to configure the stack.

The deployment uses Docker Swarm and sets up an OpenCRVS stack containing each service with a number of replicas defined in the docker compose files. Note: This deployment is currently automated so that every time we push to master the build will be deployed during the CI process.

The deploy is easily executed by just running: yarn deploy:staging --clear-data=yes --restore-metadata=yes <<insert host>> <<insert version>> - you will need ssh access to the server for this to work.

How can I deploy to a QA environment cluster?

Deploying to QA is much the same as above, however you may specify a version to deploy. The version can be any docker image tag. Each time master is build on CI docker images are created for that commit hash. Any of these hashes may be used as the version. In addition any time a git tag is created and pushed all the docker images will automatically build. Once complete the name of this tag can be used to deploy to the QA environment as well.

yarn deploy:qa --clear-data=yes --restore-metadata=yes <<insert host>> <<insert version>>

How can I deploy to production?

Deploying to Production is much the same as deploying to QA.

yarn deploy:prod --clear-data=yes --restore-metadata=yes <<insert host>> <<insert version>>

How do I export recent registrations?

You can export registrations (both death and birth) from the last n days by running a script from infrastructure/export-registrations.sh <<insert number of days>> <<insert host>> <<insert system admin token>>.

Would create a new .zip file export.zip with 2 CSV files inside of it. You can tweak the time period from inside the script if needed.

How does OpenCRVS back up registration data?

OpenCRVS backs up all of its data by default and the Ansible playbook script here allows you to configure a remote server in which all data will be zipped and copied to on a nightly cron job. Follow the instructions here

Become part of the OpenCRVS Community

We want to see OpenCRVS implemented across the world. We can’t do this alone. Through the OpenCRVS Community, we are uniting experts in civil registration and other interested parties.

Join the community

Join our Gitter community

Contributing

You may view/add issues here: https://github.com/opencrvs/opencrvs-core/issues

To contribute code, please review the CONTRIBUTING.md file https://github.com/opencrvs/opencrvs-core/blob/master/CONTRIBUTING.md, fork the repository and submit a pull request. The authors will review the code and merge it in if all is well.

By contributing to the OpenCRVS code, you are conforming to the terms of the license below.

How to tag an OpenCRVS release?

So you have contributed to core and want to make a new release as an OpenCRVS core repo admin?

1. Update all packages with the new version number according to semver. All packages will have the same version for simplicity as they are all designed to be used together. Update all dependencies to point to the newly created versions. E.g. client depend on components, so update the dependency: Do a find and replace for 1.0.0-alpha.2 and replace with 1.0.0-alpha.3

2. Run yarn to ensure there are no version errors.

3. Run yarn test and ensure all passed.

4. Run git tag v<version_number> e.g. git tag v1.0.0-alpha.3.0

5. Run git push origin v<version_number>

6. Create a new release on Github using the tag you just pushed and including any release notes.

7. Dockerhub should automatically build the images when a new release tag is created in Git. Howver Dockerhub can sometimes timeout and you may need to compose and push the release tagged images locally. To do that, run yarn compose:push:release

Is OpenCRVS live anywhere, and what functionality is currently available in the latest release?

OpenCRVS is currently released as a public Alpha, and has been deployed in 2 districts of Bangladesh in a live pilot.

A huge number of business processes have been built including end-to-end registraion of birth and death events, de-duplication, performance analytics and user management.

Throughout 2020 we will be refreshing existing functionality and releasing more user, facility and content management features in advance of a full public Beta release. Please get in touch with us to find out more about our roadmap & talk to us on Gitter.

License

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at https://mozilla.org/MPL/2.0/.

OpenCRVS is also distributed under the terms of the Civil Registration & Healthcare Disclaimer located at http://opencrvs.org/license.

Copyright (C) Plan International Inc, Plan International Australia, Jembi Health Systems NPC and Vital Strategies Inc.