martinteaching / phenoflow-ml

Antonio López Martínez-Carrasco, 2022

Portable, workflow-based phenotype definitions.

Live site | Wiki | API

Cite as Phenoflow: A Microservice Architecture for Portable Workflow-based Phenotype Definitions. Chapman, Martin et al. AMIA Summits on Translational Science. 2021.

---

Components of the Phenoflow architecture.

Prerequisites

1. Docker (machine).
2. Python 3.
3. Pip.
4. Virtualenv.

Configuration

Certificate generation

1. Run visualiser/git-server/certs/gen-ca-cert.sh.
2. Run visualiser/git-server/certs/gen-domain-cert.sh.
3. Copy visualiser/git-server/certs/pf.pem to a new folder visualiser/spring/certs.
4. Copy visualiser/git-server/certs/pf.pem to a new folder web/certs.
5. Copy visualiser/git-server/certs/pf.* to web/proxy/certs.
6. Run web/proxy/certs/gen-domain-cert.sh.

web .env (development)

1. Create a .env file in web.

2. Generate a local development RSA private key (for JWT signing), generated by openssl genrsa -out key.pem 2048, and reference its location (e.g. certs/key.pem, if the key is generated and stored within certs):

RSA_PRIVATE_KEY_FULL_PATH=

.env (production)

1. Create a .env file in project root.

2. To this file, add the following, which should be chosen at random:

DEFAULT_PASSWORD=
MYSQL_ROOT_PASSWORD=
MYSQL_USER=
MYSQL_PASSWORD=
MYSQL_DATABASE=

3. In addition, add the location of a root certificate within web (e.g. 'certs/pf.pem'), allowing 'web' to communicate with other services:

NODE_EXTRA_CA_CERTS=

4. Add a deployment RSA private key name, e.g. key.pem (openssl genrsa -out key.pem 2048):

RSA_PRIVATE_KEY=

5. Add a path on the target deployment host to the deployment RSA private key (without the key name), e.g. /home/ubunut/.certs/:

HOST_RSA_PRIVATE_KEY_PATH=

DB encryption

6. Run web/db/encrypt/keys/db-encryption.sh.

Misc

• Create analytics.pug in web/views (may be blank, or contain analytics script)
• Add images to web/public/images (live set available here)
• Add a .htpasswd file to web/proxy/certs
• Create web/uploads

Development

This is an express (lightweight server) project. The majority of the logic is contained within web/app.js, and in the web/routes and web/util folders.

Install and Run

Visualiser

• Local running is only available via Docker. From visualiser:

docker-compose -f docker-compose.dev.yml build
docker-compose -f docker-compose.dev.yml up -d

Generator

1. Inside generator, initialise a virtual environment, and activate:

virtualenv -p python3 env
. env/bin/activate

2. Install dependencies:

pip install -r requirements.txt

3. Run generator:

python main.py

Web

1. Inside web, create a node virtual environment (within a python virtual environment), and activate it:

virtualenv env
. env/bin/activate
pip install nodeenv
nodeenv nenv
. nenv/bin/activate

2. Install dependencies:

cat requirements.txt | xargs npm install -g

3. Create database:

npm run migrate

4. Run server, referencing generated certificate:

NODE_EXTRA_CA_CERTS=certs/pf.pem npm start

Deployment

1. Deployment is via Docker. If using remote machine, ensure it is activated:

eval $(docker-machine env [machine-name])

2. From the root directory, build these containers:

docker-compose build

3. Run these containers:

docker-compose up -d

4. Repeat inside visualiser.

Tests

Development

• Inside web, run all tests:

npm test

• Run specific test(s), e.g.:

NODE_ENV=test npx mocha -g "basic" --exit

• Inside generator, run all tests:

python -m unittest

Deployment

• From the root directory run:

docker-compose -f docker-compose.test.yml build;
docker-compose -f docker-compose.test.yml up;

Tests can also be used to import phenotypes from different data sources.

Usage

Development

The server runs by default on port 3003. Visit localhost:3003/[route] to test changes to GET endpoints and use software such as Postman to test changes to POST (and other) endpoints.

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Authors

kclhi

License

This project is licensed under the MIT License - see the LICENSE.md file for details.

Acknowledgments

• python-cwlgen.
• cwlviewer.