cloud-native-toolkit / terraform-gitops-cp-db2wh-service

Module to populate a GitOps repo with the resources to deploy the underlying DB2 Warehouse services from CP4D required to create a DB2 Warehouse instance

CP4D - DB2 Warehourse Gitops terraform module

DB2WH Pre-Req

Ensure that a cluster administrator completed the required Pre-installation tasks for your environment. Specifically, verify that a cluster administrator completed the following tasks:

• Cloud Pak for Data is installed.
• The cluster is configured to pull the Db2 Warehouse software images.
  • Make sure the global pull secret is applied and worker nodes are replaced.
• The db2u-operator catalog source exists.
• The db2u-operator operator subscription exists.

If these tasks are not complete, the Db2 Warehouse installation will fail.

Db2 Warehouse on Cloud Pak for Data

IBM Db2 Warehouse is an analytics data warehouse that features in-memory data processing and in-database analytics. It is client-managed and optimized for fast and flexible deployment, with automated scaling that supports analytics workloads.

3 Namespace(s) are used in this module to support special installation

• operator_namespace: cpd-operators CP4D Platform operator, DB2WH Operator will be installed on cpd-operators

• common_services_namespace: ibm-common-services CP4D foundational services, Operand Deployment Lifecycle manager Operator and IBM zen service will be installed on the ibm-common-service namespace.

• cpd_namespace: gitops-cp4d-instance DB2WH instance will be installed on gitops-cp4d-instance

Cloud Pak for Data, Db2WH Subscription and Db2WHService instance gitops module

Module to provision a gitops repo with the resources necessary to provision a Cloud Pak for data,ibm-db2WH-cp4d-operator Subscription and Db2WHService instance on a cluster. In order to provision Subscription and the instance, the following steps are performed:

1. As we are using the Specialized installations architecture(where the IBM Cloud Pak foundational services operators and Cloud Pak for Data operators are in separate OpenShift projects), we must edit the IBM Cloud Pak foundational services operand registry to point to the project where the Cloud Pak for Data operators are installed:

• Edit the IBM Cloud Pak foundational services operand registry
• Change the value of the namespace entry for the ibm-db2u-operator. Specify the OpenShift project where the Cloud Pak for Data operators are installed, for example, cpd-operators:
• Save the changes

2. Add the db2wh Subscription chart to the gitops repo (charts/ibm-cpd-db2wh-subscription)
3. Add the Db2whService instance chart to the gitops repo (charts/ibm-cpd-db2wh-instance)

Unit tests is expected to be executed on a cluster that already has CP4D-instance and its dependencies installed and configured.

Module dependencies

This module makes use of the output from other modules:

• GitOps - github.com/cloud-native-toolkit/terraform-tools-gitops.git
• Namespace - github.com/cloud-native-toolkit/terraform-gitops-namespace.git
• gitops_ibm_catalogs - github.com/cloud-native-toolkit/terraform-gitops-cp-catalogs.git
• gitops_cp_foundation - github.com/cloud-native-toolkit/terraform-gitops-cp-foundational-services.git
• gitops_cp4d_operator - github.com/cloud-native-toolkit/terraform-gitops-cp4d-operator.git
• gitops-cp4d-instance - github.com/cloud-native-toolkit/terraform-gitops-cp4d-instance.git

Suggested companion modules

The module itself requires some information from the cluster and needs a namespace to be created. The following companion modules can help provide the required information:

• Gitops: github.com/cloud-native-toolkit/terraform-tools-gitops
• Gitops Bootstrap: github.com/cloud-native-toolkit/terraform-util-gitops-bootstrap
• Namespace: github.com/ibm-garage-cloud/terraform-cluster-namespace
• Pull Secret: github.com/cloud-native-toolkit/terraform-gitops-pull-secret
• Cert: github.com/cloud-native-toolkit/terraform-util-sealed-secret-cert
• Cluster: github.com/cloud-native-toolkit/terraform-ocp-login

DB2WH Service check

Run this CLI and check if the DB2WHService completed.

• oc project gitops-cp4d-instance

• oc get Db2whService db2wh-cr -o jsonpath='{.status.db2whStatus} {"\n"}'

Db2 Warehouse is ready when the command returns "Completed".

DB2WH Service (instance) removal - Finalizer

Run this CLI and remove the finalizer value from the YAML as sometimes DB2WH service getting stuck.

• oc edit db2whservice db2wh-cr -n gitops-cp4d-instance

Supported platforms

• OCP 4.8.36

References:

• DB2 Warehouse Knowledge Center

terraform-gitops-cp-db2wh-service's Issues

Update module to support "Specialized" installation - multiple namespace

update the module to support the "Specialized" installation which will refer the common-services, cpd-opertors and cpd-instances namespace in the openshift

DB2W modules expects catalog sources are missing

as part of CP4D, the catalog sources are missings

https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=ccs-creating-catalog-sources-that-automatically-pull-latest-images-from-entitled-registry

Step 1 and Step 2 are missing

Variable names updates in chart

Need to update the variable name in the chart template values

Provide a secret that contains the credentials and connectivity information for the DB2 warehouse instance

Once DB2 warehouse has been provisioned we need the connection information for the instance. There are five pieces of information required:

• host
• port
• database name
• username
• password

The way to store/pass this information in the cluster is via a secret.

• Create a secret that contains the required information (ideally this secret already exists but if not a job is needed to create it). The host can be the internal service DNS name ({service name}.{namespace}.svc)

• Add the following outputs to the module

  • secret_name - the name of the secret that contains the connectivity information
  • host_key - the key in the secret that contains the host information (e.g. "host")
  • port_key - the key in the secret that contains the port information
  • database_key - the key in the secret that contains the database name information
  • username_key - the key in the secret that contains the username information
  • password_key - the key in the secret that contains the password information

• Add the interface definition to the module metadata

  name: gitops-cp-db2wh
  type: terraform
  interfaces:
    - github.com/cloud-native-toolkit/automation-modules#database-secret
  

Create a module to provision DB2 warehouse via gitops

Generated Terraform Template Framework needs to updated for the DB2W module with latest framework

Generated Framework needs to be updated such as ValidateDeploy.sh, Test Stages , Main.tf, verify.yaml etc to sync up with latest Terraform Template

Source Repo - https://github.com/cloud-native-toolkit/template-terraform-module

DB2WH Service deletion works sometimes or hangs forever

Based on number of test, i noticed the db2wh service deletion works or hangs. I need to remove the finalizer before i attempt to delete the db2wh service

https://ibm-cloudplatform.slack.com/archives/C019EJ0QH4Y/p1651254139179939

DB2W Namespace not getting created - Build Fail Errors

During the build, the db2w namespace not getting created as well as values.yaml not getting created which causes build to fail.

Waiting for namespace: gitops-cp-db2wh
Waiting for namespace: gitops-cp-db2wh
Waiting for namespace: gitops-cp-db2wh
Waiting for namespace: gitops-cp-db2wh
Waiting for namespace: gitops-cp-db2wh
Waiting for namespace: gitops-cp-db2wh
Waiting for namespace: gitops-cp-db2wh
Waiting for namespace: gitops-cp-db2wh
Timed out waiting for namespace: gitops-cp-db2wh
Validation failed
Error: Process completed with exit code 1.

DB2W GitOPS Build Fails

Github action failing during the build failed

./payload/2-services/namespace/cpd-operators/ibm-db2u-operator/Chart.yaml
./payload/2-services/namespace/ibm-common-services
./payload/2-services/namespace/ibm-common-services/ibm-db2oltp-cp4d-operator
./payload/2-services/namespace/ibm-common-services/ibm-db2oltp-cp4d-operator/templates
./payload/2-services/namespace/ibm-common-services/ibm-db2oltp-cp4d-operator/templates/subscription.yaml
./payload/2-services/namespace/ibm-common-services/ibm-db2oltp-cp4d-operator/templates/_helpers.tpl
./payload/2-services/namespace/ibm-common-services/ibm-db2oltp-cp4d-operator/values.yaml
./payload/2-services/namespace/ibm-common-services/ibm-db2oltp-cp4d-operator/Chart.yaml
./payload/2-services/namespace/gitops-cp-db2wh
./payload/2-services/namespace/gitops-cp-db2wh/db2warehouse
./payload/2-services/namespace/gitops-cp-db2wh/db2warehouse/db2wh_sub.yaml
./payload/2-services/namespace/gitops-cp-db2wh/db2warehouse/db2wu_sub.yaml
./payload/2-services/namespace/gitops-cp-db2wh/db2warehouse/db2wh_cr.yaml
./payload/2-services/.gitkeep
./payload/1-infrastructure
./payload/1-infrastructure/namespace
./payload/1-infrastructure/namespace/gitops-cp-db2wh
./payload/1-infrastructure/namespace/gitops-cp-db2wh/namespace
./payload/1-infrastructure/namespace/gitops-cp-db2wh/namespace/rbac.yaml
./payload/1-infrastructure/namespace/gitops-cp-db2wh/namespace/ns.yaml
./payload/1-infrastructure/.gitkeep
./payload/3-applications
./payload/3-applications/.gitkeep
./README.md
./kubeseal_cert.pem
./config.yaml
ArgoCD config missing - argocd/2-services/cluster/default/base/gitops-cp-db2wh-my-module.yaml
Validation failed
Error: Process completed with exit code 1.