openshift / ocp-release-operator-sdk

Overview

This is the downstream repo for the Operator Framework's Operator SDK. The purpose of this repo is to build downstream Operator SDK for OpenShift releases.

Documentation

The downstream documentation can be found: Downstream Docs. You can also refer to the upstream Operator SDK website.

License

Operator SDK is under Apache 2.0 license. See the LICENSE file for details.

Downstream structure

This repo is a mirror of the upstream operator-framework/operator-sdk repo plus some ci files.

The downstream ci files are as follows:

• ci - contains the CI related build files
• patches - contains the patches applied to downstream builds
• release/ansible - ansible operator release files
• release/helm - helm operator release files
• release/sdk - sdk operator release files
• vendor - unlike the upstream, we vendor in the dependencies downstream

Syncing

When an upstream release is ready, you can sync down that release downstream.

Verify upstream

Verify you have the upstream repo as a remote:

$ git remote -v
openshift       https://github.com/openshift/ocp-release-operator-sdk.git (fetch)
openshift       no_push (push)
origin  [email protected]:jmrodri/ocp-release-operator-sdk.git (fetch)
origin  [email protected]:jmrodri/ocp-release-operator-sdk.git (push)
upstream        https://github.com/operator-framework/operator-sdk.git (fetch)
upstream        no_push (push)

Ensure the upstream is up to date.

$ git fetch upstream
remote: Enumerating objects: 21, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (15/15), done.
Unpacking objects: 100% (15/15), 17.36 KiB | 2.89 MiB/s, done.
remote: Total 15 (delta 8), reused 3 (delta 0), pack-reused 0
From https://github.com/operator-framework/operator-sdk
 * e3225a40..4c0a60dc  master                                                    -> upstream/master

New release to master

Once the upstream has been verified. You can now sync the upstream release tag you want. In this steps below we will sync v1.4.1 to master.

To build simply use the UPSTREAM-MERGE.sh script.

./UPSTREAM-MEGE.sh <UPSTREAM-TAG>

Here is an example run using upstream tag v1.4.1

$ ./UPSTREAM-MERGE.sh v1.4.1
Already on 'master'
Your branch is up to date with 'origin/master'.
Already up to date.
Switched to a new branch 'v1.4.1-rebase-master'
Removing test/ansible/roles/inventorytest/tasks/main.yml
Removing test/ansible/requirements.yml
Removing ...
...
CONFLICT (file location): ...
...
Automatic merge failed; fix conflicts and then commit the result.
rm 'hack/generate/samples/internal/ansible/testdata/build/Dockerfile'
...
[v1.4.1-rebase-master f7bb26ab] Merge upstream tag v1.4.1
5338217e (tag: v1.4.1, upstream/latest) Release v1.4.1 (#4521)
5d6e333a [v1.4.x] docs/upgrade-sdk-version/v1.3.0.md: correct link to migration guide (#4517)
...

** Upstream merge complete! **
View the above incoming commits to verify all is well
(mirrors the commit listing the PR will show)

Now make a pull request.

When the script is finished, you will be in a UPSTREAM-TAG-rebase-master branch. In our example above we were in v1.4.1-rebase-master. For more information about the output see the breakdown of UPSTREAM-MERGE.sh output section below.

At this point, verify things look okay. Ensure the patches apply, if they do not you will have to either recreate the patch to make it apply, or remove it entirely if it is no longer needed. This will require manual intervention and inspection.

Once the patches have been verified, create a release PR downstream:

git push origin v1.4.1-rebase-master

Breakdown of UPSTREAM-MERGE.sh output

Running UPSTREAM-MERGE.sh will spew out a bunch of output. We'll break it down here.

The script will verify your branch is up to date and create a new branch UPSTREAM-TAG-rebase-master.

$ ./UPSTREAM-MERGE.sh v1.4.1
Already on 'master'
Your branch is up to date with 'origin/master'.
Already up to date.
Switched to a new branch 'v1.4.1-rebase-master'

Then the script will printout any files it needs to remove as part of syncing to upstream. This can be many files when doing a major or minor update with upstream. This is normal.

Removing test/ansible/roles/inventorytest/tasks/main.yml
Removing test/ansible/requirements.yml
Removing ...

If there are any conflicts, it will identify those files. It will always choose the file from the upstream tag.

CONFLICT (file location): ...
...
Automatic merge failed; fix conflicts and then commit the result.
rm 'hack/generate/samples/internal/ansible/testdata/build/Dockerfile'

After removing and conflicted files are listed, the commits from the tags are printed out:

[v1.4.1-rebase-master f7bb26ab] Merge upstream tag v1.4.1
5338217e (tag: v1.4.1, upstream/latest) Release v1.4.1 (#4521)
5d6e333a [v1.4.x] docs/upgrade-sdk-version/v1.3.0.md: correct link to migration guide (#4517)
e385e760 [v1.4.x] generate: fix multiple Go file type parsing bug (#4509)
67f9c8b8 (tag: v1.4.0) Release v1.4.0 (#4486)
26eacd81 update IMAGE_VERSION to v1.4.0 (#4484)
...

Finally you will get a completed message:

** Upstream merge complete! **
View the above incoming commits to verify all is well
(mirrors the commit listing the PR will show)

Now make a pull request.

Patch release to specific release branch

There are times when you will want to bring an upstream patch release to the downstream OpenShift release. For example, bringing down v1.3.2 to downstream release-4.7.

We will use the same script, UPSTREAM-MERGE.sh except we will add the branch we want to patch.

./UPSTREAM-MERGE.sh v1.3.2 release-4.7

Here is the output of running the above script.

$ ./UPSTREAM-MERGE.sh v1.3.2 release-4.7
Already on 'release-4.7'
Your branch is up to date with 'openshift/release-4.7'.
Already up to date.
Switched to a new branch 'v1.3.2-rebase-release-4.7'
Automatic merge went well; stopped before committing as requested
[v1.3.2-rebase-release-4.7 aebcb02a] Merge upstream tag v1.3.2
5dd883dc (tag: v1.3.2, upstream/v1.3.x) Release v1.3.2 (#4520)
73d76d0a [v1.3.x] docs/upgrade-sdk-version/v1.3.0.md: correct link to migration guide (#4516)
e4fc90c4 [v1.3.x] generate: fix multiple Go file type parsing bug (#4508)
0ec4b705 [v1.3.x] images/ansible-operator/Dockerfile: pin cryptography python package (#4510)
af8bc8dd (tag: v1.3.1) Release v1.3.1 (#4485)
835b648d [v1.3.x] bump IMAGE_VERSION to v1.3.1 (#4483)
9c178b76 [v1.3.x] generate: make CSV generator for Go GVKs package-aware (#4480)
9f7e16d2 [v1.3.x] Fixed invalid object names generated for long package names  (#4476)
16b8daee Align tutorial imports with test samples (#4465)
f340790f [v1.3.x]  Bug 1921727: Bundle validate should not fail because of warnings. (#4449) (#4458)
4bf11a29 [v1.3.x] Bug fix: helm operator uninstall is not properly checking for existing release (#4457)
ddcda902 Handle error when subscription doesn't match package name for `run bundle-upgrade` (#4454)
96970e9e [v1.3.x] Bug 1921458: `run bundle-upgrade` should handle error gracefully when a previous operator version doesn't exist (#4451)
4bee54fb generate: respect project version when getting package name (#4431) (#4443)
7f7cc9a6 [v1.3.x] (helm/v1,ansible/v1) fix download URLs and order of binary checks (#4412)
608aa050 [v1.3.x] Fix panic with "operator-sdk bundle validate" and OCI (#4386)
bebbbf98 testdata/go: regenerate with updated license date (#4388)
b5404095 docs: fix broken link (#4387)

** Upstream merge complete! **
View the above incoming commits to verify all is well
(mirrors the commit listing the PR will show)

Now make a pull request.

Just like sync to master, verify things look okay. Ensure the patches apply, if they do not you will have to either recreate the patch to make it apply, or remove it entirely if it is no longer needed. This will require manual intervention and inspection. You can manually verify the patches apply by running:

make -f ci/prow.Makefile patch build

Once the patches have been verified, create a release PR downstream:

git push origin v1.3.2-rebase-release-4.7

Building

To build this downstream repo, you should use the ci/prow.Makefile. The patches need to be applied first.

make -f ci/prow.Makefile patch build

Patching

This downstream repo tries to maintain the upstream code as original as possible. There are times when changes need to be made to make the upstream source work downstream or to react to a release blocking bugzillas before they can be merged upstream.

gendiff method

This method uses the gendiff tool which can create patches from a set of files that have a common backup extension. The gendiff tool will create a patch file for all files in the given directory that have the matching common backup extension. The patch is stored in the patches directory. They are applied sequentially during the build process.

Let's walk through an example. Assume we need to patch cmd/operator-sdk/main.go & cmd/ansible-operator/main.go.

1. create backup copies of each file with a specific extension. Note, the extension is used to group the changed files into a single patch.

   cp cmd/operator-sdk/main.go cmd/operator-sdk/main.go.bugxxx
   cp cmd/ansible-operator/main.go cmd/ansible-operator/main.go.bugxxx
   

2. make the necessary changes to the original files

   vi cmd/operator-sdk/main.go
   vi cmd/ansible-operator/main.go
   

3. once you are done create the patch using gendiff

   gendiff . .bugxxx > patches/07-bugxxx.patch
   

4. add the patch to the patches directory

   git add patches/07-bugxxx.patch
   

5. commit the new patch

   git commit --signoff
   

6. revert the changed original files

   git restore cmd/operator-sdk/main.go
   git restore cmd/ansible-operator/main.go
   

7. test the patch applies

   make -f ci/prow.Makefile patch
   # verify both main.go files were patched
   

8. restore the changed files again and push up your changes

   git restore cmd/operator-sdk/main.go
   git restore cmd/ansible-operator/main.go
   git push origin bugxxx
   

ocp-release-operator-sdk's Issues

UBI9/RHEL9 ansible-operator based images

Feature Request

Describe the problem you need a feature to resolve.

The currently supplied base images for ansible-operator are still all RHEL8 based , many products are moving their base images to rhel9 including ours but there isn't a matching ubi9/rhel9 build available.

[fbladilo@fdev:~/release/cpaas/mta/versions/release-6.2.0] (relocate_operator =)$ docker run --rm -it --entrypoint /bin/bash registry.redhat.io/openshift4/ose-ansible-operator:latest

bash-4.4$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.6 (Ootpa)
bash-4.4$

Describe the solution you'd like.

A RHEL9 based ansible-operator to be supplied along with the existing rhel8 release.

/language ansible

Future Release Branches Frozen For Merging | branch:release-4.16 branch:release-4.17

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.16
• release-4.17

For more information, see the branching documentation.

bundle: nice UX around disconnected setup

Feature Request

Describe the problem you need a feature to resolve.

A bundle's CSV supports "related image" configuration at the catalog level, which effectively establishes a pointer from some static image label to a resolved (i.e. contains the image's remote digest) image tag. Such config is crucial for disconnected clusters due to image proxying.

Related image config can be done via .spec.relatedImages (see #operator-framework/operator-sdk#4884) or setting a RELATED_IMAGE_{name} environment variable. The latter is recommended, since OSBS will populate .spec.relatedImages from env vars matching the aforementioned pattern.

Describe the solution you'd like.

A kustomize configuration can "teach" filters to apply transformers to particular field sets. The following setup will substitute images set with kustomize edit set image key=value into .spec.relatedImages and RELATED_IMAGE_* env vars with the right key:

# config/manifests/relatedimages_config.yaml
images:
# For .spec.relatedImages
- path: spec/relatedImages[]/image
  create: true
# For env vars
- path: spec/install/spec/deployments[]/spec/template/spec/containers[]/env[]/value
  create: true

# config/manifests/kustomization.yaml
...
configurations:
- relatedimages_config.yaml

# This transformer makes sure the deployment name is set with the right prefix.
# The resulting value will be important for CSV parsing by 'operator-sdk generate bundle'.
transformers:
- |-
  apiVersion: builtin
  kind: PrefixSuffixTransformer
  metadata:
    name: notImportantHere
  prefix: memcached-operator-
  fieldSpecs:
  - path: spec/install/spec/deployments[]/name
    create: false
    kind: ClusterServiceVersion
    group: operators.coreos.com
    version: v1alpha1

Note: You would not actually want both .spec.relatedImages and RELATED_IMAGE_* configuration options present because the latter will not overwrite the former. I put forth both in the same issue because discussion is needed on which to support.

Given the below base CSV, you can run kustomize edit set image controller=foo.com/bar@sha256:1234 to create a kustomization.yaml image transformer that maps reference controller to foo.com/bar@sha256:1234. When you run kustomize build config/manifests, you will see a CSV with the expected image reference updates.

config/manifests/bases/memcached-operator.csv.yaml contents

apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    alm-examples: '[]'
    capabilities: Basic Install
  name: memcached-operator.v0.0.0
  namespace: placeholder
spec:
  apiservicedefinitions: {}
  customresourcedefinitions:
    owned:
    - description: Memcached is the Schema for the memcacheds API
      displayName: Memcached
      kind: Memcached
      name: memcacheds.cache.example.com
      version: v1alpha1
  description: Memcached Operator description. TODO.
  displayName: Memcached Operator
  icon:
  - base64data: ""
    mediatype: ""
  install:
    spec:
      deployments:
      - name: controller-manager
        spec:
          template:
            spec:
              containers:
              - name: manager
                env:
                - name: RELATED_IMAGE_FOO_BAR
                  value: controller
                - name: SOME_ENV
                  value: foo
    strategy: deployment
  installModes:
  - supported: false
    type: OwnNamespace
  - supported: false
    type: SingleNamespace
  - supported: false
    type: MultiNamespace
  - supported: true
    type: AllNamespaces
  keywords:
  - memcached-operator
  links:
  - name: Memcached Operator
    url: https://memcached-operator.domain
  maintainers:
  - email: [email protected]
    name: Maintainer Name
  maturity: alpha
  provider:
    name: Provider Name
    url: https://your.domain
  version: 0.0.0
  relatedImages:
  - image: controller
    name: foo_bar

@@ -23,17 +23,17 @@
   install:
     spec:
       deployments:
-      - name: controller-manager
+      - name: memcached-operator-controller-manager
         spec:
           template:
             spec:
               containers:
               - name: manager
                 env:
                 - name: RELATED_IMAGE_FOO_BAR
-                  value: controller
+                  value: foo.com/bar@sha256:1234
                 - name: SOME_ENV
                   value: foo
     strategy: deployment
   installModes:
   - supported: false
@@ -56,7 +56,7 @@
   provider:
     name: Provider Name
     url: https://your.domain
   version: 0.0.0
   relatedImages:
-  - image: controller
+  - image: foo.com/bar@sha256:1234
     name: foo_bar

To support env vars, some changes are needed in operator-sdk generate bundle, namely the ability to merge RELATED_IMAGE_* variables into the correct Deployment. This should not be difficult given Deployments and their containers have unique names in this context.

ansible-runner version in openshift4/ose-ansible-operator tag: v4.12.0 version has a security vulnerability

Bug Report

A flaw was found in ansible-runner. An improper escaping of the shell command, while calling the ansible_runner.interface.run_command, can lead to parameters getting executed as host's shell command. A developer could unintentionally write code that gets executed in the host rather than the virtual environment.
This problem is solved in ansible-runner version 2.1.0

Please update the version of ansible-runner in the Ansible Operator Base Image (openshift4/ose-ansible-operator)

Future Release Branches Frozen For Merging | branch:release-4.5 branch:release-4.4

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.5
• release-4.4

Contact the Test Platform or Automated Release teams for more information.

Add requirements for building downstream

Downstream requires patch command to be installed to build. There are other requirements those should be identified and listed in the README as well.

Re-introduce ci directory which was removed upstream

Feature Request

Is your feature request related to a problem? Please describe.
The ./ci/ directory is used only in this repository in tandem with OpenShift CI, so we removed it upstream in operator-framework/operator-sdk#2483

Describe the solution you'd like
Next time we sync this repository from upstream, the ./ci directory will disappear. We should revert operator-framework/operator-sdk@9ffaefc in this repository after syncing from upstream.

Future Release Branches Frozen For Merging | branch:release-4.8 branch:release-4.9

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.8
• release-4.9

Contact the Test Platform or Automated Release teams for more information.