U.S. Forest Service Open Forest Migration and Configuration

Overview

This migration script is to port over the U.S.F.S. Open Forest system to a new organization within cloud.gov, or presumably another cloud foundry instance. It consists of three applications with code in two separate repositories.

The repositories are the [https://github.com/18f/fs-open-forest-platform] and the [https://github.com/18f/fs-open-forest-middlelayer-api].

Each application is deployed in development, staging, and production environments housed in separate corresponding spaces within the cloud.gov org. For more on cloud.gov orgs and spaces, please see the cloud foundry docs

This migration script does not provide any data held in the s3 buckets or databases.

This repo may also help you manage the connected services of the application, bound within VCAP_SERVICE in the environment variables.

Overview
Table of contents
Migration Requirements
Environment Variables
Migration
After Migration
- Update CI Keys
- Generate Middlelayer Users
Updating Environment Variables
Utilities
Notes
Contributing
Public Domain

Migration Requirements

Bash

All scripts in this repository are written using BASH. If you are using MacOS, Linux, Unix, this will be available in your default shell. If you are using Windows, please contact your IT support to determine the best way to run BASH scripts.

Cloud Foundry CLI

CF CLI

Cloud Foundry Plugins

JQ

Environment variables

Prior to running the migration, you will need a local copy of the environment variables for each application.

User provided services

The user provided services a cloud foundry feature to help manage external 3rd party services. This ecosystem makes heavy use of them in order to manage the credentials of integrated services.

These user provided services are parsed as environment variables in a vcap-constant file in both the platform definitions and middlelayer services. While each repository will give a better indication of the services use, here are some of the expected values and how to generate them.

For more on why we use then instead of environment variables.

Organization

These should be organized in a group of json files, one per application:

json-envs/intake/intake-services-dev.json
json-envs/intake/intake-services-staging.json
json-envs/intake/intake-services-production.json
json-envs/middlelayer/middlelayer-services-dev.json
json-envs/middlelayer/middlelayer-services-staging.json
json-envs/middlelayer/middlelayer-services-production.json

Format

Each file should contain an array of the required services and credentials in the format below. See VCAP_SERVICES for more information.

[
  {
    "name": "servicename1",
    "credentials": {
  	  "key1": "anytype",
  	  "key2": "anytype"
  	}
  },
  {
    "name":"servicename2",
    "credentials" : {
      "key3": "anytype",
      "key4": "anytype"
  },
  ...
]

Values

For details on the required values see the documentation in the respective repository for each application:

Fetching

To obtain all environment variables programmatically from existing Cloud Foundry infrastructure:

Make sure you are in the root of this repository and have installed of the requirements
Log into the correct org with the Cloud Foundry CLI Ex. cf login -a api.fr.cloud.gov -o usda-forest-service --sso
./bin/get-credentials.sh -- Note this will overwrite existing local files so save a copy if necessary.

Migration

./migration.sh

If you would like to not run the migration tasks and just create the new apps and corresponding services within an org run:

./migration.sh false

After Migration

Update CI Keys

For each environment, a set of "deployer credentials" is generated for use in other systems that need to interact with Cloud[.]gov programmatically. Circle CI is configured with these credentials in order to trigger deploys to staging and production environments on the successful completion of automatic tests. In addition, certain other credentials, such as those for additional S3 buckets, are generated in Cloud[.]gov to leverage their brokered services. If any of these change, they will need to be updated in the Circle CI user interface as well:

Generate Middlelayer Users

See Middlelayer for instructions on how to generate a user. After a user is generated, the configuration for the 'platform' in the corresponding environment will have to be updated with the generated authentication credentials. See Updating Environment Variables for details.

Updating environment variables

When credentials change, the environment variables in Cloud[.]gov will need to be updated and the corresponding application restaged. For the example below, we will assume a credential changed for the staging intake application. WARNING these changes are IRREVERSABLE so make sure to use the appropriate Cloud[.]gov space, application, and env variable file.

Update your local copy by following the instructions here
Update the value in json-envs/<env variable file>.json
cf t -s <space>
cf multi-cups-plugin -p json-envs/<env variable file>.json
cf restage <application>
where
- env variable file: intake/intake-services-staging
- space: open-forest-platform-staging
- application: open-forest-platform-staging

Utilities

For our vendor integrations we are required to generate and provide certificates in order to sign and/or encrypt the transactions. We provide some useful scripts and instructions to help with this process.

Utilities Requirements

OpenSSL

OpenSSL

Node

Node (Prefer installing using a version manager such as NVM)

X509 Certificate Generation

Login[.]gov and EAuth require X509 certificates to integrate. Generate a unique public certificate and private key for each integration and environment:

Create a configuration file named openssl-<integration-environment>.conf in the certificates/conf folder where "integration-environment" is one of:
- login-tree (-> Open Forest staging)
- login-prod
- eauth-dev (-> Open Forest staging)
- eauth-prod
Populate the configuration file with the following:

  [ req ]
  default_bits           = 2048
  distinguished_name     = req_distinguished_name
  prompt                 = no

  [ req_distinguished_name ]
  commonName             = <API_APP_URL>
  organizationName       = USDA
  organizationalUnitName = FORESTSERVICE
  localityName           = Washington
  stateOrProvinceName    = DC
  countryName            = US
  emailAddress           = [email protected]

where "API_APP_URL" is the URL of the application.

run ./certificate-generation/make.sh <integration-environment> where "integration-environment" corresponds to the configuration file

This will generate the following files:
- ./certificates/keys/saml.key.enc.usda-forest-service-epermit-<integration-environment>
- ./certificates/certs/saml.crt.usda-forest-service-epermit-<integration-environment>
If necessary, create a Json Web Key (JWK) from the private key by following the steps in JWK Generation.
The appropriate public certificates in certificates/certs should be shared with the Eauth/Login[.]gov while the private keys (or corresponding JWK) will be used in our application configuration.

JWK Generation

The Login[.]gov integration requires that the private key be provided in the form of a Json Web Key or JWK.

npm install
node certificate-generation/jwkmaker.js ./certificates/keys/saml.key.enc.usda-forest-service-epermit-<integration-environment>"

Public Key Certificate Generation

Pay[.]gov requires a Public Key Certificate to integrate. This section is not verified...

obtain paygov.pfx FROM Pay[.]gov?

run openssl pkcs12 -in paygov.pfx -out outfile.pem -nodes

The resulting file will have the cert, the leveraged certificate certs and the private key. The certificate will need to be an array of the certificates as strings in the following format:

"certificate": [
        "-----BEGIN CERTIFICATE-----\ncert1-----END CERTIFICATE-----\n",
        "-----BEGIN CERTIFICATE-----\ncert2-----END CERTIFICATE-----\n",
        "-----BEGIN CERTIFICATE-----\ncert3-----END CERTIFICATE-----\n",
        "-----BEGIN CERTIFICATE-----\ncert4-----END CERTIFICATE-----\n",
        "-----BEGIN CERTIFICATE-----\ncert5-----END CERTIFICATE-----\n"
      ],

The private key should be added to the private_key in the user-provided service. The password used to generate will need to be added as well.

Notes

cf ssh is currently disabled for the production Cloud[.]gov middlelayer-production and public-production spaces.

Contributing

See CONTRIBUTING for additional information.

Public Domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

isabella232 / fs-cloud-gov-migration Goto Github PK

fs-cloud-gov-migration's Introduction