sap / component-operator-runtime

A framework that supports creating opinionated Kubernetes operators managing the lifecycle of arbitrary deployment components of Kubernetes clusters.

Kubernetes Component Operator Runtime

About this project

A framework to support development of Kubernetes operators managing Kubernetes components.

The operators implemented through this framework are strongly opiniated in the sense that instances of the managed component are described through a dedicated, specific custom resource type.

The key features are:

• Efficient and smart handling of Kubernetes API extensions (custom resource definitions or API aggregation).
• Ability to fully take over rendering of the component's resources by implementing

    type Generator interface {
        Generate(ctx context.context, namespace string, name string, parameters types.Unstructurable) ([]client.Object, error)
    }

  (where parameters correspond to the spec of the describing custom resource component object).
• Projects having existing Helm charts or Kustomizations describing the component's lifecylce can reuse these by bundling them into the component operator, leveraging the included ready-to-use HelmGenerator resp. KustomizeGenerator implementations.
• Scaffolding tool to bootstrap new component operators in minutes, see the Getting Started documentation.

Documentation

The project's documentation can be found here: https://sap.github.io/component-operator-runtime.
The API reference is here: https://pkg.go.dev/github.com/sap/component-operator-runtime.

Support, Feedback, Contributing

This project is open to feature requests/suggestions, bug reports etc. via GitHub issues. Contribution and feedback are encouraged and always welcome. For more information about how to contribute, the project structure, as well as additional contribution information, see our Contribution Guidelines.

Code of Conduct

We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone. By participating in this project, you agree to abide by its Code of Conduct at all times.

Licensing

Copyright 2023 SAP SE or an SAP affiliate company and component-operator-runtime contributors. Please see our LICENSE for copyright and license information. Detailed information including third-party components and their licensing/copyright information is available via the REUSE tool.

component-operator-runtime's Issues

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(actions): update actions/configure-pages action to v5
• fix(major deps): update node.js to v22

Detected dependencies

dockerfile

scaffold/templates/Dockerfile.tpl

github-actions

.github/workflows/publish-scaffold.yaml

• actions/checkout v4
• actions/setup-go v5
• ubuntu 22.04

.github/workflows/publish-website.yaml

• actions/checkout v4
• actions/configure-pages v4
• actions/setup-node v4
• actions/upload-pages-artifact v3
• actions/deploy-pages v4
• ubuntu 22.04

.github/workflows/publish.yaml

• ubuntu 22.04

.github/workflows/release.yaml

• actions/checkout v4
• ubuntu 22.04

.github/workflows/run-tests.yaml

• actions/checkout v4
• actions/setup-go v5
• actions/checkout v4
• actions/setup-go v5
• actions/checkout v4
• actions/setup-go v5
• ubuntu 22.04
• ubuntu 22.04
• ubuntu 22.04

gomod

go.mod

• go 1.22.3
• github.com/Masterminds/sprig/v3 v3.2.3
• github.com/hashicorp/go-multierror v1.1.1
• github.com/iancoleman/strcase v0.3.0
• github.com/onsi/ginkgo/v2 v2.17.3
• github.com/onsi/gomega v1.33.1
• github.com/pkg/errors v0.9.1
• github.com/prometheus/client_golang v1.19.1
• github.com/sap/go-generics v0.2.11
• github.com/spf13/pflag v1.0.5
• golang.org/x/time v0.5.0
• k8s.io/api v0.30.1
• k8s.io/apiextensions-apiserver v0.30.1
• k8s.io/apimachinery v0.30.1
• k8s.io/client-go v0.30.1
• k8s.io/kube-aggregator v0.30.1
• sigs.k8s.io/cli-utils v0.36.0
• sigs.k8s.io/controller-runtime v0.18.2
• sigs.k8s.io/kustomize/api v0.17.1
• sigs.k8s.io/kustomize/kyaml v0.17.0
• sigs.k8s.io/structured-merge-diff/v4 v4.4.1
• sigs.k8s.io/yaml v1.4.0

helmv3

internal/helm/testdata/main/Chart.yaml

internal/helm/testdata/main/charts/sub11/Chart.yaml

npm

website/package.json

• node 21.7.3
• autoprefixer ^10.4.13
• postcss ^8.4.21
• postcss-cli ^11.0.0

---

• Check this box to trigger a request for Renovate to run again on this repository

scaffolded helm chart: add podmonitor/prometheusrule (optional)

The scaffolded helm chart (./scaffold/templates/chart) should be able to optionally render a PodMonitor object such as

{{- if .Values.metrics.podMonitor.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: {{ include "xxx.fullname" . }}
  labels:
    {{- include "xxx.labels" . | nindent 4 }}
spec:
  namespaceSelector:
    matchNames:
    - {{ .Release.Namespace }}
  selector:
    matchLabels:
      {{- include "xxx.selectorLabels" . | nindent 6 }}
  podMetricsEndpoints:
  - port: metrics
{{- end }}

and a PrometheusRule object such as

{{- if and .Values.metrics.prometheusRule.enabled .Values.metrics.prometheusRule.rules }}
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: {{ include "xxx.fullname" . }}
  labels:
    {{- include "xxx.labels" . | nindent 4 }}
spec:
  groups:
  - name: {{ include "xxx.name" . }}
    rules:
    {{- toYaml .Values.metrics.prometheusRule.rules | nindent 4 }}
{{- end }}

Deployments do not get updated to the new version during upgrade

Hi Colleagues,

We noticed that in some cases, the cap-operator version doesn't get updated to the new version when we upgrade the cap-operator-manager. It seems to happen randomly and is not consistently reproducible.

Our test scenario -

1. We have the cap-operator-manager running on version 1.0.7 which will deploy the cap-operator version 1.0.7.
2. We created a CRO and it deployed the cap-operator with version 1.0.7 into the namespace as expected.
3. Now we triggered an update by deploying version 1.0.8 of cap-operator-manager which should ideally deploy cap-operator version 1.0.8. But the cap-operator didn't get updated. We tried waiting for some time but still, the version was not updated.
4. But on restarting the cap-operator-manager pod, the cap-operator gets updated to the new version 1.0.8.

We were able to capture this behavior in a recording - Link. Let me know if you have any issue accessing the recording.

Please find the logs below just after updating cap-operator-manager to version 1.0.8 -

{"level":"info","ts":"2023-08-01T12:07:39Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"} {"level":"info","ts":"2023-08-01T12:07:39Z","logger":"setup","msg":"starting manager"} {"level":"info","ts":"2023-08-01T12:07:39Z","msg":"Starting server","kind":"health probe","addr":":8081"} {"level":"info","ts":"2023-08-01T12:07:39Z","msg":"starting server","path":"/metrics","kind":"metrics","addr":":8080"} I0801 12:07:39.458809 1 leaderelection.go:245] attempting to acquire leader lease cap-operator-system/cap-operator.sme.sap.com... I0801 12:07:48.909261 1 leaderelection.go:255] successfully acquired lease cap-operator-system/cap-operator.sme.sap.com {"level":"info","ts":"2023-08-01T12:07:48Z","msg":"Starting EventSource","controller":"capoperator","controllerGroup":"operator.sme.sap.com","controllerKind":"CAPOperator","source":"kind source: *v1alpha1.CAPOperator"} {"level":"info","ts":"2023-08-01T12:07:48Z","msg":"Starting Controller","controller":"capoperator","controllerGroup":"operator.sme.sap.com","controllerKind":"CAPOperator"} {"level":"info","ts":"2023-08-01T12:07:49Z","msg":"Starting workers","controller":"capoperator","controllerGroup":"operator.sme.sap.com","controllerKind":"CAPOperator","worker count":3}

Logs after restarting the cap-operator-manager pods-

{"level":"info","ts":"2023-08-01T12:08:29Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"} {"level":"info","ts":"2023-08-01T12:08:29Z","logger":"setup","msg":"starting manager"} {"level":"info","ts":"2023-08-01T12:08:29Z","msg":"Starting server","kind":"health probe","addr":":8081"} {"level":"info","ts":"2023-08-01T12:08:29Z","msg":"starting server","path":"/metrics","kind":"metrics","addr":":8080"} I0801 12:08:29.774471 1 leaderelection.go:245] attempting to acquire leader lease cap-operator-system/cap-operator.sme.sap.com... I0801 12:08:29.797858 1 leaderelection.go:255] successfully acquired lease cap-operator-system/cap-operator.sme.sap.com {"level":"info","ts":"2023-08-01T12:08:29Z","msg":"Starting EventSource","controller":"capoperator","controllerGroup":"operator.sme.sap.com","controllerKind":"CAPOperator","source":"kind source: *v1alpha1.CAPOperator"} {"level":"info","ts":"2023-08-01T12:08:29Z","msg":"Starting Controller","controller":"capoperator","controllerGroup":"operator.sme.sap.com","controllerKind":"CAPOperator"} {"level":"info","ts":"2023-08-01T12:08:29Z","msg":"Starting workers","controller":"capoperator","controllerGroup":"operator.sme.sap.com","controllerKind":"CAPOperator","worker count":3} I0801 12:08:32.260244 1 request.go:696] Waited for 1.004779362s due to client-side throttling, not priority and fairness, request: GET:https://api.i325261-test.cs-trial.internal.canary.k8s.ondemand.com:443/apis/rbac.authorization.k8s.io/v1?timeout=32s

After the pod restart, we see an additional log - 12:08:32.260244 1 request.go:696] Waited for 1.004779362s due to client-side throttling, not priority and fairness, request: GET:https://api.i325261-test.cs-trial.internal.canary.k8s.ondemand.com:443/apis/rbac.authorization.k8s.io/v1?timeout=32s. But not sure if this is related to the issue.

Can you please help us with this? Also, let me know if you need the cap-operator-manager deployment files. I can share them via mail.

Thanks and regards,
Anirudh

helm generator: error with named templates which call lookup

There is an error in the template rendering logic of helm/kustomize generator: whenever

• a named template (as defined by define) calling lookup is referenced by include
• an inline template calling lookup is called by tpl
  an error occurs because the client it uses to communicate with the cluster is nil.

incorrect merging when mixing create/update and ssa

How to reproduce

Create an object with update policy to ssa-merge or ssa-override, having two annotations set. Then remove one of the annotations, and wait until component-operator-runtime has updated the object.

Expected behaviour: the removed annotation is removed from the object in the cluster.

Actual behaviour: the removed annotation is still there.

Additional information

It seems that this is because (when using ssa), the framework uses Create() to create the object. As a consequence, the fields are owned by manager reconcilerName and operation Update. The ssa update then happens with Patch(), which leads to field owner records with the same manager, but with operation Apply. Which the API server treats as independent owners. As the removed annotation is not part of the specified manifest, the API server concludes that the sender of this manifest has no opinion on the removed annotation, and therefore keeps it with the old owner (the one with operation Update).