redhat-cop / declarative-openshift

Working examples of manifests for openshift for use in a declarative management strategy.

Declarative OpenShift

Looking for old stuff?

Are you looking for older content that used to exist here? If so, please check out the v1.0.0 release.

Overview

This repository contains sets of example resources to be used with a declarative management strategy. Please familiarize yourself with the terminology in that document before reading on.

Kubernetes Cluster API

Starting with the 2.x version of this repositry, the focus is on using Cluster API paired with a varity of tools to support the full declarative approach for OpenShift Cluster provisioning / bootstrapping, upgrades and multi-cluster & multi-account management.

CAPA - Cluster API Provider AWS

For managing clusters hosted on AWS, including Red Hat OpenShift on AWS (ROSA), we utilize CAPA as it nicely integrates and works together with CAPI.

OpenShift GitOps - ArgoCD

Red Hat OpenShift GitOps may now play a role in the entire lifecycle of a cluster, and the entire deployment. Not only can be it be used to manage the content of an operational OpenShift cluster, but it can also be used as part of a management cluster setup to manage the targeted clusters mananged with CAPI.

ClusterResourceSet

As part of bootstrapping a new environment, there is a need to inject new content to allow for connectivity and initial setups for connecting tooling to talk to the cluster. This can be achieved in multiple ways - including running OpenShift GitOps in the cluster for a full content management, or using ClusterResourceSet to manage a minimal set of content in the even that an external content management system is used (e.g.: running OpenShift GitOps at a centralized management cluster).

Managing OpenShift Clusters

Prerequisites

Before applying any of the manifests to create a new cluster(s), the following prerequisites need to be met:

• A valid AWS account with full admin access
• A valid account for Red Hat Hybrid Cloud Console
• Steps 1-3 here to create the necessary VPC configuration and account and operator roles.

NOTE: Steps 1-3 from above will soon be replaced with declarative content within this repo. More to come on this soon...

Management Cluster

In order to deploy a new OpenShift Cluster using CAPI, you will need a management cluster with the necessary CAPI and CAPA deployments in an operational state.

To get started, install the following tools on your local machine:

• AWS CLI
• clusterawsadm CLI tool
• clusterctl CLI tool
• kubectl and/or oc CLI tool
• helm

Run the following commands to prepare the environment and management cluster:

  export KUBECONFIG=...
  export AWS_REGION=...
  export AWS_ACCESS_KEY_ID=...
  export AWS_SECRET_ACCESS_KEY=...
  export AWS_SESSION_TOKEN=... # (optional)
  export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)
  export EXP_ROSA="true"
  export EXP_MACHINE_POOL="true"
  clusterctl init --infrastructure aws

Apply customizations provided by this repo:

Optional: This step is needed if your mangement cluster is an OpenShift Cluster

  helm template --release-name rosa-hcp charts/openshift-management | oc apply -f -

Make sure the CAPI pods are operational before running the next apply.
This can be done by checking the output of the following commands:

  oc get pods -n capi-system
  oc get pods -n capa-system
  oc get pods -n capi-kubeadm-bootstrap-system
  oc get pods -n capi-kubeadm-control-plane-system

Apply CAPI specific configurations needed to support the upcoming workloads:

  helm template charts/capi-management | oc apply -f -

Run the following command to validate that everything is set up correctly, and ready for your first ROSA HCP cluster deployment with CAPI (all commands should return output containing the values in the grep part of the command):

  oc get crd | grep rosa
  oc get deployment -n capa-system capa-controller-manager -o yaml | grep ROSA=true
  oc get deploy capi-controller-manager -n capi-system -o yaml | grep MachinePool=true

Quickstart - Simple CAPI / CAPA

Install

With the above steps from the Management Cluster done, it is now time to deploy the first (workload) cluster. For this first simple quickstart, that is done in two steps:

1. Tweak the values in charts/rosa-capi/values.yaml
2. run helm template --release-name rosa-hcp --set ocmToken=jSDFsdfsdfSDF charts/rosa-capi | oc apply -f -

NOTE: Replace the rosa-hcp release name and OCM Token above with your own values - alternatively set your OCM Token in the values file. The OCM Token can be fetched from https://console.redhat.com/openshift/token/rosa.

Validation steps

Check the status of the the ROSAControlPlane deployment:

  oc get rosacontrolplane <name-of-cluster>

Once the cluster shows state "READY" as "true", the cluster is available to access. Extract the kubeconfig and use it access the cluster:

  oc get secrets <cluste-rname>-kubeconfig -o jsonpath='{.data.value}' | base64 -d > /tmp/rosa-capi.kubeconfig
  export KUBECONFIG=/tmp/rosa-capi.kubeconfig
  oc get nodes

Troubleshooting

coming soon...

declarative-openshift's Issues

Use GitOps to build out CAPI Management Cluster

Create a Helm Chart to install cross-plane to the management cluster

Create declarative IAM content for ROSA Operator (and account) roles

Manage node labels and taints

Documentation and example using the resource-locker-operator to declaratively manage node labels and taints

In response to the following request being closed: openshift/machine-config-operator#845

Integrate ACM as part of environment bootstrapping

Create a Helm Chart to install OpenShift GitOps to the Management Cluster

Manage ROSA MachinePool through GitOps

Use GitOps to update a running Cluster

This is mainly a testing task to ensure changing existing clusters work

Ability to generate a valid prune-whitelist for existing cluster & bootstrap repo.

In order to use oc apply --prune with an OpenShift cluster, we need to extend the set of APIs that the --prune flag will act on. By default it is a static set of Kubernetes core apis. Right now we are handling this statically in a text file in the repo, but it would be better to generate a list of <Group/Version/Kind>s based on the cluster we are targetting and the set of resources we are managing.

Create, Manage, and Link secrets

Creating secrets should be fairly straightforward, but we don't have many examples of declaratively defining links from secrets to serviceaccounts. The classic example is creating a source secret and linking it to a builder serviceaccount.

Use GitOps to upgrade a target cluster

this is mainly a testing task to ensure that changing the version of a control plane and/or machine/node pool works

Draft an architecture diagram about how does this Declarative OpenShift work.

I create a initial draft about this architecture diagram. please refer to https://drive.google.com/file/d/1za2bl5TZs3vYB1521zi0UmTT7L_vPtMm/view?usp=sharing

Namespace management for CAPI

Build a structure around how namespaces are managed in CAPI Cluster creation. Currently, the namespace is set to default.

Configuring custom ingress certificates and CA trusts

Use GitOps to bootstrap environments (for management Cluster)

Use GitOps to manage the surrounding components + bootstrap clusters

Consider using a different key than example.com/project

Introduce ACK to manage AWS components

Provision OpenShift GitOps as part of cluster bootstrapping

Provision OpenShift GitOps as part of cluster bootstrapping using Helm AddOn to Cluster API addon provider for helm CAAPH[]

Use GitOps to delete a target cluster

Should be mainly a testing task

Deploy and Configure Logging Stack

This should include a custom config of Fluentd. In order to keep this self-contained, the suggested customization would be to stand up a second ES instance for audit logs, and configure the Fluentd stack to send audit logs there instead of to the primary ES instance

Integrate ACS as part of the environment bootstrapping

Document how to handle CRD/CR race condition

There is a race condition frequently created when deploying operators via OLM, where attempting to create the OperatorGroup and Subscription in the same oc apply command where the CR for that operator is also created, then the CRD for the operator hasn't yet been created. Since we don't directly create the CRD, we need to insert a conditional wait to ensure that the CRD exists before moving on.

It would also be nice if we could automate the detection of the CRDs we are waiting on, so we don't have to hardcode them in the workflow.

Create declarative ClusterResourceSet content to bootstrap content

https://cluster-api.sigs.k8s.io/developer/architecture/controllers/cluster-resource-set

Manage Operators (through OperatorHub)

We need an example of an deploying an operator and configuring its CRs to "make it do something". We'll need to handle the race condition where we can't try to apply the CR before the CRD has been created by OLM. We should be able to use the status of the Subscription or an annotation on the Subscription to check that.

Fold in ArgoCD examples to argocd-bootstrap

https://github.com/sabre1041/argocd-examples

Manage Namespaces

several things to address here:

• using the namespace configuration operator to create a namespace based on user creation (sandbox use case)
• creating the namespace based on a group (application use case)
• how to apply additional configs to a namespace:
  • quotas & limits
  • custom rbac
  • networkpolicies

We also want to ensure proper labels are placed on the namespace to represent the owner (group or user)

Edge Case: Labeling existing namespaces

There are certain scenarios, such as when setting up ingress network policies, where a label must be added to the default namespace.

Argo CD basic example failed creating basic-example

I am testing the creation of ArgoCD instance on OCP 4.5, and I got this error:
The ArgoCD "example-argocd" is invalid: spec.server.route: Invalid value: "boolean": spec.server.route in body must be of type object: "boolean"

It turned out from the CD that the CRD is looking for an object instead of a boolean on the new operator version.

Edge Case: Infra Nodes or otherwise adding labels to existing nodes

I labelled this as an edge case because we might not be able to do this declaratively. But there might be a way.

A Multi-Instance Database Deployment

Customers often ask about this one. Should be able to use an Operator.

Introduce Crossplane to manage Cloud Infrastructure Components

Manage Security Context Constraints

We should include an example of how to configure an SCC. We should also deploy some sort of privileged container in a namespace that we can use to prove escalated privileges.

Provision OpenShift GitOps as part of cluster bootstrapping to make it available for applications

Potentially paired with #41 - we'd want OpenShift GitOps to be ready immediately as soon as the cluster is operational.

Authentication & Group Sync

Provision ROSA cluster using a CrossPlane Composition

Develop a cross-plane XRD and a Composition that enables the deployment of a ROSA cluster through a single Helm chart, eliminating the need for manual steps.

Cluster Role Based Access Control

Need examples of managing RBAC. Two things we should include:

1. A custom role of some kind, possibly using aggregate-to.
2. a ClusterRoleBinding to a user or group