bcgov / cas-pipeline

A collection of make functions used to compose pipelines

cas-pipeline

A set of shell scripts, makefiles and a helm chart to help the Climate Action Secretariat team deploy applications to the OpenShift cluster.

Intended usage

The Makefile in this repository has two main commands used to, respectively, grant access to the appropriate OpenShift namespaces, and to add common configuration items in said namespaces

make authorize

To be run anytime the CAS team members change, or new namespaces are added

Loads the list of namespaces from the .env file (see .env-example, the actual .env file is stored in the teams password manager), reads the list of users present in the appropriate GitHub teams (see teams descriptions on GitHub), and create a RoleBinding object for each user/namespace pair.

This script first deletes the previously created RoleBindings (identified by a label) and recreates them based on the GitHub teams membership, to ensure that previous team members access is revoked.

Because it temporarily revokes access for all team members, this needs to be run manually by one of the current members listed as technical lead on the project registry, as they are also granted access via a RoleBinding created by the Platform Services team.

make provision

Deploys the [cas-provision helm chart] to every namespace used by the team. This relies on a hidden .values.yaml file (stored in the team's password manager) and creates various objects such as:

• deployer, linter and job-runner roles following the least privilege principle. Those roles are only allowed to manage objects used in the CAS team's projects, and would need to be updated if new OpenShift object types were to be used (e.g. see PR #53, when horizontalpodautoscalers were first used).
• secrets listing the namespace names of the various applications (which are randomly generated when provisions), providing easier referencing for developers, .i.e. an application can look at the cas-namespaces secret to refer to the namespace of other applications.
• a DockerHub registry credential
• a SysdigTeam object, which is a custom resource created by platform services to grant access to the Sysdig monitoring platform.
• various secrets containing credentials used by our applications
• Utilizing gcp (the Google Cloud Platform CLI), creates buckets for TF state for every namespace used by the team. Relies on a being authorized with a service account (credentials stored in the team's password manager) with storage permissions on the project.
  • Note: gcp will give errors when a bucket is created already under the service accounts control. The script ignores these errors, as they don't need to block further buckets from being created or the rest of the make target executing.

make install_crunchy_monitoring

Deploys the [crunchy-monitoring helm chart] to the namespace defined in the values file. This relies on a hidden .crunchy-values.yaml file (stored in the team's password manager).

• requires defining the CIIP_NAMESPACE_PREFIX variable

make lint_monitoring_chart

Lints the [crunchy-monitoring helm chart]

• requires defining the CIIP_NAMESPACE_PREFIX variable

Adding a namespace

Deprecated things

Prior to using Helm to deploy applications to the OpenShift cluster, the CAS team used a set of common make commands (e.g. configure, build, install) that abstracted the oc command line tool. These came with various utility functions located in the *.mk files, which are still in use in some projects but are considered deprecated.

Terraform in CAS repos

Usage

1. Import the Helm Chart into your project's main chart as a dependency.
2. Update your values.yaml (and any environmental versions of values) with those required by the terraform-bucket-provision chart:

   2a. If the project shares a namespace with another one (as is the case with cas-metabase sharing cas-ggircs's namespace), use the workspace value with anything other than default to create a seperate Terraform workspace in the state to avoid overwriting.

terraform-bucket-provision:
  terraform:
    namespace_apps: '["example-project-backups", "example-project-uploads"]'
    workspace: example # This value is OPTIONAL, only set if required

Components

~/helm/terraform-bucket-provision/

This repo contains a Helm chart that contains a job that will import and run Terraform files. It deploys at the pre-install, pre-upgrade hooks. This chart references secrets and config that is deployed to a namespace when a project is provisioned by cas-pipeline (credentials, project_id, kubeconfig, terraform backendconfig).

terraform-apply.yaml: This file defines the Job that deploys a container to run Terraform. Secrets (deployed by make provision) contain the credentials and .tfbackend Terraform uses to access the GCP buckets where it stores state. The terraform-modules.yaml ConfigMap is what pulls in the Terraform scripts that will be run.

~/helm/terraform-bucket-provision/terraform

In tandem with the Helm chart is a Terraform module that creates GCP storage buckets, service accounts to access those buckets (admins and viewers) and injects those credentials into OpenShift for usage. These modules are pulled in via a configMap which pulls all files from this charts /terraform directory. These are bundled with the chart as the way we use Terraform is currently identical in our CAS projects.

cas-pipeline's Issues

Configure access to the OpenShift CIF namespaces

CAS Devops team

Update maintainers and members of CAS devops team

Add missing topics

TL;DR

Topics greatly improve the discoverability of repos; please add the short code from the table below to the topics of your repo so that ministries can use GitHub's search to find out what repos belong to them and other visitors can find useful content (and reuse it!).

Why Topic

In short order we'll add our 800th repo. This large number clearly demonstrates the success of using GitHub and our Open Source initiative. This huge success means its critical that we work to make our content as discoverable as possible; Through discoverability, we promote code reuse across a large decentralized organization like the Government of British Columbia as well as allow ministries to find the repos they own.

What to do

Below is a table of abbreviation a.k.a short codes for each ministry; they're the ones used in all @gov.bc.ca email addresses. Please add the short codes of the ministry or organization that "owns" this repo as a topic.

That's in, you're done!!!

How to use

Once topics are added, you can use them in GitHub's search. For example, enter something like org:bcgov topic:citz to find all the repos that belong to Citizens' Services. You can refine this search by adding key words specific to a subject you're interested in. To learn more about searching through repos check out GitHub's doc on searching.

Pro Tip 🤓

• If your org is not in the list below, or the table contains errors, please create an issue here.

• While you're doing this, add additional topics that would help someone searching for "something". These can be the language used javascript or R; something like opendata or data for data only repos; or any other key words that are useful.

• Add a meaningful description to your repo. This is hugely valuable to people looking through our repositories.

• If your application is live, add the production URL.

Ministry Short Codes

Short Code	Organization Name
AEST	Advanced Education, Skills & Training
AGRI	Agriculture
ALC	Agriculture Land Commission
AG	Attorney General
MCF	Children & Family Development
CITZ	Citizens' Services
DBC	Destination BC
EMBC	Emergency Management BC
EAO	Environmental Assessment Office
EDUC	Education
EMPR	Energy, Mines & Petroleum Resources
ENV	Environment & Climate Change Strategy
FIN	Finance
FLNR	Forests, Lands, Natural Resource Operations & Rural Development
HLTH	Health
FLNR	Indigenous Relations & Reconciliation
JEDC	Jobs, Economic Development & Competitiveness
LBR	Labour Policy & Legislation
LDB	BC Liquor Distribution Branch
MMHA	Mental Health & Addictions
MAH	Municipal Affairs & Housing
BCPC	Pension Corporation
PSA	Public Safety & Solicitor General & Emergency B.C.
SDPR	Social Development & Poverty Reduction
TCA	Tourism, Arts & Culture
TRAN	Transportation & Infrastructure

NOTE See an error or omission? Please create an issue here to get it remedied.

Tech Debt: Audit GCP storage-admin and storage-viewer Service Account templates

We use two service account permission templates for the service accounts we use to work with cloud storage in GCP, storage-admin and storage-viewer. These templates should be reviewed to ensure we only allow access to permissions required, especially the admin which can be overly permissive.

Acceptance Criteria

In Our cas storage Google cloud project:

• Audit the Storage Viewer role template and make changes where needed
• Audit the Storage Admin role template and make changes where needed (creating a new one if required)

Pipeline documentation curation

Assess the CIIP repo's documentation.
Things to consider:

• What is outdated
• What is missing
• Is it organised in a way that makes sense and is discoverable

Todo:

• Remove or update outdated documentation
• Create a list of suggestions for documentation that should be added
• Re-organise documentation if it is not discoverable following our documentation organisation strategy

Documentation organisation strategy:

• A docs directory at the root of a repo
• Separate doc files with names that clearly describe what the doc file contains
• Appropriately named sub-directories where there are several related individual doc files

Create artifactory service account in every namespace

Description

To reduce the costs and time of downloading images from gcr.io, we should use Artifactory as a cache.
The artifactory documentation can be found here: https://github.com/BCDevOps/developer-experience/blob/master/apps/artifactory/DEVHUB-README.md

The first step to be able to download images through artifactory is to create an account per namespace.

It's Been a While Since This Repository has Been Updated

This issue is a kind reminder that your repository has been inactive for 180 days. Some repositories are maintained in accordance with business requirements that infrequently change thus appearing inactive, and some repositories are inactive because they are unmaintained.

To help differentiate products that are unmaintained from products that do not require frequent maintenance, repomountie will open an issue whenever a repository has not been updated in 180 days.

• If this product is being actively maintained, please close this issue.
• If this repository isn't being actively maintained anymore, please archive this repository. Also, for bonus points, please add a dormant or retired life cycle badge.

Thank you for your help ensuring effective governance of our open-source ecosystem!

Move artifactory service account out of cas-provision helm chart

It seems that Helm will always try to patch the ArtifactoryServiceAccount object, even if there are not actual changes. Patching of ArtifactoryServiceAccount is not permitted as it can lead to undesirable behaviour in the operator. Unless we can find how to avoid Helm attempting to patch the ArtifactoryServiceAccount for no reason, the solution will be to move it to a different helm chart, and to create a make provision_artifactory target that we only use once.

The new service account must keep the same name, so that helm chart using artifactory don't need updating

However, deleting and re-creating the service account will lead to a new randomly-named pull secret being generated, which means that we will need to redeploy the last deployment of all of the applications using artifactory (airflow and cif), so that Helm can find the name of the new pull secrets.

make_install_crunchy monitoring should create a list of alert managers from a github team

The list of people to be alerted by our crunchy monitoring is hardcoded in the .values file. It would be better to get these values from a github team. (There is an example of this in the same makefile for make authorize. This would make it less manual to add/remove people from the alert list.

Add project lifecycle badge

No Project Lifecycle Badge found in your readme!

Hello! I scanned your readme and could not find a project lifecycle badge. A project lifecycle badge will provide contributors to your project as well as other stakeholders (platform services, executive) insight into the lifecycle of your repository.

What is a Project Lifecycle Badge?

It is a simple image that neatly describes your project's stage in its lifecycle. More information can be found in the project lifecycle badges documentation.

What do I need to do?

I suggest you make a PR into your README.md and add a project lifecycle badge near the top where it is easy for your users to pick it up :). Once it is merged feel free to close this issue. I will not open up a new one :)

create documentation on how to use the `make provision` target on cas-pipeline

Right now there is no documentation about the helm chart we created and how it's used by the make provision target.
We need some documentation that lists:

• provide a .env and .values.yaml files (download from 1password or create)
• list what it does where, roughly
• when to call it

test issue 5

Refactor terraform chart to be more general

The Terraform-job-based chart that provisions storage buckets and accounts could be refactored to accommodate terraform broadly. It currently contains the terraform scripts that are used, but we could expand it to include values that would supply these files instead, if we want to have more use cases for terraform. As all of our projects currently use terraform in the same way, this is not needed, but could be useful for the future.

logs should be fetched for failed jobs

right now we only fetch job logs while the pod is running
https://github.com/bcgov/cas-pipeline/blob/master/lib/oc_run_job.sh#L50

if it never ran, we should still fetch the job logs

oc_validate should error if a parameter that is used is not defined in the parameters array

For instance, the following template currently passes oc_validate, but it should fail as the ${FOO} parameter is undefined:

apiVersion: template.openshift.io/v1
kind: Template

objects:
- apiVersion: image.openshift.io/v1
  kind: ImageStream
  metadata:
    name: ${FOO}
  spec:
    lookupPolicy:
      local: true

detect missing jq

since jq was introduced in #7 it was not given its own environment detection variable and error like eg. awk

Make authorize revokes script caller's permissions

When calling make authorize, the first step is to revoke all rolebindings created by the cas-pipeline helm chart. This includes the user calling the script, which results in their permissions being revoked. Once their permissions are revoked, the script cannot continue and crashes.

To fix:

• Exclude the user calling the script when executing the delete rolebinding command

tech debt: convert Helm Terraform job templates into Chart

Dev Story:
I want a helm chart for the Terraform job
because it currently exists as templates across multiple jobs
and this will help me to DRY up the code and keep it in a single repo for maintainability.

Acceptance criteria:

1. Terraform job templates unified into a chart
   • terraform-apply.yaml
   • terraform-modules.yaml
   • terraform-role-binding.yaml
   • terraform-service-account.yaml
   • terraform-role.yaml
2. Replace the existing templates in cas-registration, cas-reporting, cif, ciip, ggircs, and airflow with the above chart imported.

tech-debt: Update documentation for terraform chart

Tech debt

This card will be used to track any potential improvements that can be made to the terraform-bucket-provision chart. This chart is identified in the readme at the Terraform in CAS repos header.

Actions

• Note character limit of 13 for each namespace-app value.
• Note name of the helm chart needing import (terraform-provision-storage), as there are multiple charts in the repo.

oc_new_project should not require credentials

Now that the ubi8 images are publicly published, we don't need the io-redhat-registry secret at all times, so there should be a way for us to provision a new project on our local cluster without configuring the private registry credentials for this project.

It's Been a While Since This Repository has Been Updated

This issue is a kind reminder that your repository has been inactive for 368 days. Some repositories are maintained in accordance with business requirements that infrequently change thus appearing inactive, and some repositories are inactive because they are unmaintained.

To help differentiate products that are unmaintained from products that do not require frequent maintenance, repomountie will open an issue whenever a repository has not been updated in 180 days.

• If this product is being actively maintained, please close this issue.
• If this repository isn't being actively maintained anymore, please archive this repository. Also, for bonus points, please add a dormant or retired life cycle badge.

Thank you for your help ensuring effective governance of our open-source ecosystem!

Tech Debt: make terraform chart default the workspace to the release name

Currently, the default workspace is default, so if someone uses the chart in a namespace where another release is already using this chart, the terraform state will be overridden, unless the workspace value is changed.

This would make the default behaviour fool proof to that mistake, but requires all the existing releases to be switched over

Add directions and script for migration off of Terraform Cloud

I want move the migration script used in cas-cif to cas-pipeline along with it's directions
because we will be reusing the same pattern and script on other repos
and this will help me to remove our reliance on Terraform Cloud.

Acceptance criteria:

• migration script removed from cas-cif.
• migration script added to cas-pipeline.
• directions for use added to cas-pipeline.

Consideration and notes

If the time allows, it would be better to have a make file to preform all (or as many) of the steps required. But only if the time spent in that is less than just following the directions for each subsequent repo.