3scale-ops / saas-operator Goto Github PK

With #144, a new workload reconciler has been implemented and autossl is a good fit for it.

/kind feature
/priority important-longterm
/assign

Why?

How?

Add a new apicast controller with operator-sdk binary, extracting all yamls used on staging environment.

Requirements

• Add apicast controller with all required objects in a single CRD

Acceptance Criteria

N/A

Notes

N/A

Why?

EchoAPI was the first workload to have marin3r managed with saas-operator. Lately with Backend and Apicast, we changed the way we manage marin3r annotations thrpugh the CR, so these changes need to be applied also to EchoAPI controller, so all workloads use the same framework to configure similar things.

How?

Apply Backend marin3r config from #19 to EchoAPI controller

Requirements

• Update EchoAPI marin3r configuration

Acceptance Criteria

N/A

Notes

N/A

With #144, a new workload reconciler has been implemented and system is a good fit for it.

/kind feature
/priority important-longterm
/assign

Information of zync-que jobs is not showing the data per job type (showing Value), because the used label as grafana legend is label exported_job which don't exist:

Actually, this is weird, normally having a label exported_whatever is caused because using a reserved prometheus label into app metric labels (which is wrong from the app point of view), so when prometheus scrape the app, adds the preffix exported_X and replace the original label, to differenciate from label job (prometheus target job).

It seems it has ben renamed to job_name at PR 3scale/zync#283 (it is needed to be checked other possible changes that affects the dashboard)

Move quay.io/3scaleops/saas-operator-bundle to quay.io/3scale. Release workflow will required modifications.

Following issue #55, now is the moment to convert current backend controller from ansible to go, following the new reconciler provided by https://github.com/redhat-cop/operator-utils , which has been already used for Apicast, MappingService and AutoSSL controllers at #57

Why?

We need to add more 3scale components to saas-operator as new operator controller (ansible role). Zync entire stack has been deployed in staging, and is ready to be encapsulated in the operator.

How?

Add a new zync controller with operator-sdk binary, extracting all yamls used on staging environment: deployments, services, podmonitors, secretdefinitions, and grafana dashboards.

By the moment prometheusrules will be out of the operator scope, as they can be very dynamic, still figuring out best alerts, thresholds...

Requirements

• Add zync controller with all required objects in a single CRD

Acceptance Criteria

• Znyc deployed by the saas-operator is fully functional

Notes

N/A

OperatorSDK v0.x has been deprecated and superseded by v.1.x that introduces several breaking changes. As part of the OLM publishing #42, this issue will tackle the migration to v1.x prior to the OLM bundling.

Implement a mechanism, usable by any controller, that causes Deployment rollouts on Secret or ConfigMap changes.

The implementation will be:

• Add a label to the Deployment for each Secret/ConfigMap used which has the hash of the Secret/ConfigMap contents.
• The label will change when the data contined in the Secret/ConfigMap changes, triggering a rollout.

It will be first implemented/tested for MappingService controller, which uses a Secret as a source for an environment variable.

Why?

After deploying system onto OCP, several improvements to the 3scale SaaS Operator had been discussed.

How?

By adding all the required functionalities to the system workload (and the others if required).

Requirements

• Bump Grace Period from 30s to 60s
  • System components
  • Other workloads?
• Change rollout behaviour to scale using a custom % of the pods
  • System components
  • Other workloads?
• Change HPA behaviour to scale using a custom % of the pods
  • System components
  • Other workloads?
• Change PDB to allow rollout behaviour if required
  • System components
  • Other workloads?
• Allow custom image for the system console
• Add twemproxy sidecar for the system console

Acceptance Criteria

• Bump Grace Period from 30s to 60s
  • System components
  • Other workloads?
• Change rollout behaviour to scale using a custom % of the pods
  • System components
  • Other workloads?
• Change HPA behaviour to scale using a custom % of the pods
  • System components
  • Other workloads?
• Change PDB to allow rollout behaviour if required
  • System components
  • Other workloads?
• Allow custom image for the system console
• Twemproxy sidecar for the system console

Notes

/kind feature
/priority important-soon
/assign

Why?

We need to add more 3scale components to saas-operator as new operator controller (ansible role). System entire stack has been deployed in staging, and is ready to be encapsulated in the operator.

How?

Add a new System controller with operator-sdk binary, extracting all yamls used on staging environment: deployments, services, podmonitors, secretdefinitions, and grafana dashboards.

Requirements

• Add System controller with all required objects in a single CRD

Acceptance Criteria

• System deployed by the saas-operator is fully functional

Notes

N/A

System is the only controller from saas-operator who don't include a dashboard.

Use the product dashboard from 3scale-operator as a source to implement the system saas-operator.

Why?

Soon we are going to migrate current boxy layer in EC2 (sentinel + twemproxy) to OCP, and we need to have good serviceability around it:

• Recenly we have added the option of canary deployments on multiple workloads, we will use this feature while we migrate boxy EC2 to sentinel+ twemproxy sidecars in OCP, having both current and new pods managing traffic at the same time
• Recently a sentinel controller was added to saas-operator, so sentinel is deployed in OCP and need observability (metrics, dashboard, alerts)
• Recently a twemproxy controller was added to saas-operator, so a twemproxy sidecar can be injected on selected deployments that neeed to connect to backend-storage (backend and system deployments), so we need observability around them (metrics, dashboard, alerts)

How?

Both sentinel and twemproxy has already dashboards and alerts scrapped from EC2 by the prometheus-ec2 deployment, but now we need to convert what we have to be deployed by operator, doing lot of required changes due to the difference in the way are deployed

So it is needed to:

• Add canary dashboard to current backend dashboard (so we can see the difference in performance of both backend main and canary deployment receiving same but splitted production traffic). That way we can see if main or canary has better or worst performance before doing a full switcj to sentinel+twemproxy in OCP
• Add sentinel dashboard. Right now the sentinel dashboard we have is very simple, only monitoring the last ping reply, but now we have lots of interesting sentinel-redis metrics, as well as pod cpu/mem... sentinel metrics. So a new cmplete dashboard need to be done
• Add twemproxy dashboard. Right now twemproxy dashboard uses EC2 labels as dashboard vars, as well as EC2 cpu/mem from node_exporter. Right now twemproxy deployed as a sidecar has diffrent lables to separate between types of twemproxy type(backend/system), and we have pod metrics for cpu/mem (actually, container metrics from a pod with multiple containers)

Requirements

• Scrape sentinel metrics from the operator
• Scrape twemproxy metrics from sidecars
• Add backend canary metrics to current backend grafana dashboard
• Add sentinel grafana dashboard
• Add twemproxy grafana dashboard
• Convert legacy sentinel alerts to new alerts (maybe add new ones)
• Convert legacy twemproxy alerts to new alerts (maybe add new ones)

Review and list differences between the 3scale/saas-operator requirements and the actual 3scale/3scale-operator.

While working on https://github.com/3scale/infrastructure/pull/1011 I have seen that there are a lot of backen-listener internal metrics that are missing on backend dashboard from saas-operator, but appears on the official operator 3scale/3scale-operator#473

So it is needed to add them

With #144, a new workload reconciler has been implemented and cors-proxy is a good fit for it.

In addition, with #150, SecretDefinition can be now replaced with ExternalSecret, so it will be the first workload using the new External Secrets Operator (to be deployed in production clusters yet).

/kind feature
/priority important-longterm
/assign

Why?

At the moment we are using temporarily a OpenShift Route to publish Backend listener service. Now it is the moment to remove it, and publish the service with a AWS NLB (attaching an EIP created with TF via service annotation), and add marin3r envoy sidecar in front of backend-listener deployment, doing TLS termination, and doing ratelimit at infra level (by the moment only by IP, X-forwarded-proto header).

How?

Add maybe a new backend-listener-gateway service (or replace current one) of type LoadBalancer, AWS NLB annotation, annotations to configure cross zone load balancing, annottion to configure proxy protocol enabled via aws-nlb-helper-operator , and configure marin3r status label and free-form annotations on backend listener deployment (everything like (like echoapi).

Requirements

• Remove current backend Route from operator (route, default vars, CRD, docs..)
• Add marin3r to backend listener
• Add backend-listener service CR fields to finetune NLB
• Add CRD validation of new fields
• Add documentation and examples of new fields

Acceptance Criteria

N/A

Notes

N/A

Until now, we had separated twemproxy instances for Backend and System.

• System always uses the twemproxy-slaves-rw (slave readonly=no)
• While Backend uses the twemproxy-master.

However, upon a master failover, slaves are not ready during a few minutes, and so for a while System needs to contact twemproxy-master to make possible contact with Backend redis (not ideal, but not bad for a few minutes during a failover which happens rarely).

In the legacy platform, the twemproxy-slave-rw EC2 instances had a custom metric reporting that situation, so although being a slave-rw instance, they have configured a master or a slave-readonly-yes. These are the associated prometheus alerts:

      - alert: BackendStorageBoxySlaveWithMaster
        expr: boxy_is_slave{boxy_role="slave"} < 1
        for: 5m
        labels:
          severity: critical
        annotations:
          message: "Backend redis storage instance {{ $labels.slave }} in {{ $labels.instance }} it's a master. Servers on {{ $labels.instance}} should be slaves, not masters"

      - alert: BackendStorageBoxySlaveWithSlaveReadOnly
        expr: boxy_read_only_slave{boxy_role="slave"} > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          message: "Backend redis storage instance {{ $labels.slave }} in {{ $labels.instance }} it's a slave read_only=yes. Servers on {{ $labels.instance }} should be slaves with read_only=no"

So, in case of these alerts firing, oncall is paged telling us that this twemproxy-slave-rw used by System is missconfigured and need proper manual configuration, telling us with labels which is the wrong associated shard and the wrong associated redis-server configured.

https://github.com/3scale/infrastructure/blob/master/backend_layer/docker/redis-backend/files/usr/local/sbin/boxy_prometheus.sh

However, right now with the new approach of TwemproxyConfig and twemproxy sidecars, we have lost this alert.

In the peace of code at TwemproxyConfig generator

In case of having a twemproxy-slaves-rw TwemproxyConfig, if there are no available slaves-rw, twemproxy config will point to master. On next reconcile period (every 30 seconds), it will be checked again, and once slaves-rw gets available again (which normally require manual human intervention to update roles, so can take a few minutes to happen), twemproxy-slave-rw will do a fallback (will stop traffic to master and point to slaves-rw again).

However, there is no prometheus metric telling us we are in the state of a twemproxy-slave-rw using masters, which is very useful, because upon a failover, we normally need to manually update redis roles (1 master, 1 slave-rw-no, and 1 slave-rw-yes), and can take between 15 and 30 minutes (if we forgot about it, thanks to the alert we know it needs to be done)

To not lose an important metric we have in the legacy twemproxy environment, we should have a counter metric reporting that a twemproxy-slave-rw is using a master, so that way we can add an alert with a ratefunction that will page oncall in case it happens for more than X minutes.

        - alert: TwemproxySlave
          expr: rate(saas_twemproxy_slave_rw_using_master[1m]) > 0
          for: 1m
          labels:
            severity: critical
          annotations:
            message: "Twemproxy slave is pointing to a redis master instead a slave-rw!!!"

Having System pointing to master is dangerous, from time to time System does redis slow queries of more than a minute doing sorts, which blocks the whole redis shard, that why System ideally should always point to slaves-rw.

The system-sidekiq-default with current value:

-q critical -q backend_sync -q events -q zync,40 -q priority,25 -q default,15 -q web_hooks,10 -q deletion,5

Is not loading correctly the argument by System-sidekiq (note the Queues field):

Because it should be an array of strings separated by space, we can use something like strings.Split().

In addition, the go code refer to it as QueuesArg (singular), while the CRD field is called queuesArgs (plural), as the CR field is not yet used on any CR, we should use either one or the other, so we might just update the CR field to queuesArg.

Is your feature request related to a problem? Please describe.

We want to publish saas-operator on OperatorHub.io through OLM, so anyone can easily deploy it.

Describe the solution you'd like

Follow https://operator-framework.github.io/olm-book/ instructions to make this operator OLM "compliant" and make a pull request to include in the Community operators marketplace: https://github.com/operator-framework/community-operators/blob/master/docs/contributing.md

Describe alternatives you've considered

Manual installation using all the Kubernetes objects required by the operator (deployment, rbac, ...).

Additional context

Good example of operator contribution: operator-framework/community-operators#1934

With #144, a new workload reconciler has been implemented and echo-api is a good fit for it.

/kind feature
/priority important-longterm
/assign

Why?

Operators everywhere ...

How?

Add a new mapping-service controller with operator-sdk binary, extracting all yamls used on staging environment.

Requirements

• Add mapping-service controller with all required objects in a single CRD

Acceptance Criteria

N/A

Notes

N/A

While working on sentinel grafana dashboard #197 I discovered a couple of bugs:

Bug 2: standard metrics being reported ad infinitum where a role changes

All standard metrics are retrieved from sentinel commands.

For every redis-server on a given shard with a given role, there is a given timeseries database (tuple with 3 elements)

However, when there is any change of the role from a given redis-server, it is a created a new timeseriesDB for the new shard--role2--redis-server, however the old shard--role1--redis-server timeseriesDB keeps being reported with latest value, although this tuple does not exist anymore.

Example

This redis server 10.65.6.5 from shard01 was a master long time ago (in yellow) but it is now a slave (blue)

• So the yellow line is fake, is a flat line with the latest retrived value once this server was a master in the past
• The correct one is the blue, showing it is a slave, while last ping replied has this specific sawtooth graph

Another example is the role reported time:

• Theoretically, here we should see 6 redis-servers instances reporting their role (2 shards, 3 instances per shard)
• However there have been a few failovers, with redis-servers changing their role within the same shard, and now there are 9 active timeseries
• For example, this role reported time is something should grow every milisecond until role changes (which normally not happens often)
• There are a couple of timeseries reporting constantly a few seconds (44s and 14s), this means there were 2 consecutive failovers, where a couple of redis-server occupied a given role for only 44s and 14s, but these 2 metrics are still being reported although they lasted just a few seconds

And this apply to any metric where the role label is added in https://github.com/3scale-ops/saas-operator/blob/8aafa688780f86dcb013bf8bf7fe884e1bf44d43/pkg/redis/metrics/sentinel_metrics.go

Workaround

A workaround would be to remove the role label from every metric, so the timeseries would incldue the tuple of shard-redis-server (instead of shard--role--redis-server), however I think being sentinel metrics it make sense to always have this role label.

Ideal solution

IMO, the ideal solution would be to stop reporting metrics whose tuple shard--role--redis-server in not active anymore.

• So on a given case of a redis-server x.x.x.x from shard1, passing from master to slave role, what we would should see is:
  • First reported metric shard1--master--x.x.x.x
  • A failover occurs
    -shard1--master--x.x.x.x stops being reported (is not active anymore), and now shard1--slave--x.x.x.x is the reported (active)
  • On a graph we would easily vew that redis-server x.x.x.x* was a master and now is a slave

Why?

High Availability configuration (HorizontalPodAutoscaler, PodDisruptionBudget, podAntiaffinity) was added to Backend controller on ISSUE #8 (PR #29).

Now it is the moment to add the same features to the rest of the components.

How?

Repeat what it was done on #29 regarding HA (HorizontalPodAutoscaler, PodDisruptionBudget, podAntiaffinity):

• Adding PDB and HPA templates
• Adding default values for PDB/HPA
• Adding replicas ignore and podAntiaffinity to Deployments
• Add CRD documentation and full-example
• Update CRDs

To the rest of controllers:

• Apicast
• Zync
• AutoSSL
• CORSProxy
• MappingService
• EchoAPI

It needs to be added to all deployment that can 3scale (for example, backend-cron just needs a single pod, so it is not needed), the same applies for example to system-sphinx.

Requirements

• Add HA (HorizontalPodAutoscaler, PodDisruptionBudget, podAntiaffinity) to Apicast
• Add HA (HorizontalPodAutoscaler, PodDisruptionBudget, podAntiaffinity) to Zync
• Add HA (HorizontalPodAutoscaler, PodDisruptionBudget, podAntiaffinity) to AutoSSL
• Add HA (HorizontalPodAutoscaler, PodDisruptionBudget, podAntiaffinity) to CORSProxy
• Add HA (HorizontalPodAutoscaler, PodDisruptionBudget, podAntiaffinity) to MappingService
• Add HA (HorizontalPodAutoscaler, PodDisruptionBudget, podAntiaffinity) to EchoAPI

Acceptance Criteria

N/A

Notes

N/A

We are in the process of switching from SecretDefinitons (from secrets-manager) to ExternalSecrets (external-secrets operator).

We need to update our saas-operator to create this new ExternalSecrets custom resources instead of current SecretDefinitons.

We will need to update current CRDs by adding to the fromVault spec:

• Optional cluster-store (default value vault-mgmt, so no need to update current CRs with this new field)
• Optional refreshInterval (default value 60s, so no need to update current CRs with this new field)
• Removing from the vault path field the string secrets/data/ if set (not needed on ExternalSecrets), so no need to update current CRs with this existing field
  That way current CRs will still work and change will be transparant.

Example, from:

apiVersion: secrets-manager.tuenti.io/v1alpha1
kind: SecretDefinition
metadata:
  name: system-database-seed
spec:
  name: system-database-seed
  type: Opaque
  keysMap:
    DB_NAME:
      path: secret/data/kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      key: DB_NAME
    DB_USER:
      path: secret/data/kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      key: DB_USER
    DB_PASSWORD:
      path: secret/data/kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      key: DB_PASSWORD
    URL:
      path: secret/data/kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      key: URL

To:

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: system-database-seed
spec:
  refreshInterval: 60s
  secretStoreRef:
    name: vault-mgmt
    kind: ClusterSecretStore
  target:
    name: system-database-seed
  data:
  - secretKey: DB_NAME
    remoteRef:
      key: kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      property: DB_NAME
  - secretKey: DB_USER
    remoteRef:
      key: kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      property: DB_USER
  - secretKey: DB_PASSWORD
    remoteRef:
      key: kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      property: DB_PASSWORD
  - secretKey: URL
    remoteRef:
      key: kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
      property: URL 

As the target secrets will remain the same (the secret.spec.data won't change, so won't be triggered any deployment), to avoid the target secrets being removed temporarily and recreated by external-secrets-operator, it will be needed to:

• Set secrets-manager operator to 0 pods
• Deploy new saas-operator version
• ExternalSecret will be created by saas-operator, and will adopt current target secret without doing changes on the secrets data (if it is configured correctly), which is the one used by saas-operator hashes:
  • It will add owner references to target secret (you can check it on argo now, an ExternalSecret resource which creates a Secret resource
  • Target secret labels will be merged with ExternalSecrets propagated labels and current ones from secrets-manager (last updated date, and updatedBy secrets-manager)
• SecretDefinition won't be deleted, you need to manually edit each SecretDefinition by removing the metadata.finalizers field, then the SecretDefinition can be deleted
• Delete manually secret-manager labels last updated date and updatedBy secrets-manager from target secrets
• Set secrets-manager operator to 1 pod
• Target secrets won't be updated anymore by secrets-manager

Why?

We need to add more 3scale components to saas-operator as new operator controller (ansible role). One of the main and more complex components (in terms of number of objects involved...) is 3scale backend, so it is the next controller to be implemented, reducing the complexity of application management to a single and small CR (to handle differences between environements with CR fields...).

How?

Add a new backend controller with operator-sdk binary, extracting all yamls used on staging environment: deployments, services, route, configmap, podmonitors, secretdefinitions, and grafana dashboards (backned main app dashboard, and all dashboards related to storage on EC2, nodes, redis, sentinel, twemproxy...).

By the moment prometheusrules will be out of the operator scope, as they can be very dynamic, still figuring out best alerts, thresholds... (so there is no need to update operator image with a new release to add, finetune, delete alerts or apply small changes).

Requirements

• Add backend controller with all required objects in a single CRD

Acceptance Criteria

N/A

Notes

N/A

While working on sentinel grafana dashboard #197 I discovered a couple of bugs:

Bug 1: special counter metrics

Most of the sentinel metrics are retrieved from a few sentinel commands, and are gauges.

However there are 3 counter metrics which are the most important ones defined at https://github.com/3scale-ops/saas-operator/blob/8aafa688780f86dcb013bf8bf7fe884e1bf44d43/pkg/redis/events/watcher.go:

• switch-to-master counter (one per shard): indicates there is a failover
• sdown-counter (one per redis-server): indicates a redis-server is subjectively down by a given sentinel (once all sentinel arrives to a quorum, then it is objectively marked as down)
• abort-failover-no good-slaves counter (one per shard): a failover needs to happen because master is not responding, but there are not good slaves. We dont want to arrive to this situation

The thing is, being a counter, we need to use prometheus rate function, to see increases over time (1 minute for example):
rate(saas_redis_sentinel_sdown_count{}[1m])

So something like (indicating 2 consecutive failovers on a given shard):

However:

• This metric is not reported if since the operator pod starts, if eveything works OK and there is no failover or no sdown events, being null (not zero).
• When there is an event (like sdown because we have stopped an slave instance), the counter goes from not existing to value 1, which means that rate function is 0:
  
  

So:

• This 3 metrics (2 for every shard, 1 for every redis-server) should always return a 0 once saas-operator starts
• If there is an event, increment the counter, so prometheus rate function will go from 0 to 1, and can be used on both dashboard and alerts

On e1ea6e6 it was update sentinel SubjectiveDown metric label, from server to redis_server, to use same convention as all the other sentinel metrics.

Don't know if it was introduced a bug, or that directly rem.target.ip is the sentinel pod ip (range 10.130.X.X). while we want the label redis_server to be filled with each monitored redis instance IP (range 10.65.X.X).

This is causing, that together with a descheduler problem (restarting sentinel pods with constant evictions), to report SDOWN alerts costantly because sentinel pods are recreated with diferent IPs (but of course using same hostname as it is an statefulset).

While working on some grafana dashboards improvements, I removed manually a grafana dashboard managed by my local saas-operator, and the dashboard was not being recreated, reporting error "unable to get: /slopez because of unknown namespace for the cache" (slopez was my namespace):

2021-03-02T10:48:56.654+0100	ERROR	resource-reconciler.slopez/example.integreatly.org/v1alpha1/GrafanaDashboard/slopez/backend	unable to retrieve 	{"namespace": "e.Meta.GetNamespace()", "error": "unable to get: /slopez because of unknown namespace for the cache"}
github.com/go-logr/zapr.(*zapLogger).Error
	/home/slopez/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*resourceModifiedPredicate).Delete
	/home/slopez/go/pkg/mod/github.com/redhat-cop/operator-utils@v1.1.1/pkg/util/lockedresourcecontroller/resource-reconciler.go:213
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
	/home/slopez/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/source/internal/eventsource.go:131
k8s.io/client-go/tools/cache.(*processorListener).run.func1
	/home/slopez/go/pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/shared_informer.go:779
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/home/slopez/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/home/slopez/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/home/slopez/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
	/home/slopez/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run
	/home/slopez/go/pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/shared_informer.go:771
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
	/home/slopez/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:73

How to reproduce it

I executed the operator locally watching a specific namespace:

make run WATCH_NAMESPACE=slopez

On a different terminal, I created a sample Backend CR on the watched namespace:

$ cat backend.yaml
apiVersion: saas.3scale.net/v1alpha1
kind: Backend
metadata:
  name: example
  namespace: slopez
spec:
  image:
    tag: v3.2.0
  config:
    rackEnv: dev
    redisStorageDSN: backend-redis-storage
    redisQueuesDSN: backend-redis-queues
    systemEventsHookURL:
      fromVault:
        key: URL
        path: secret/data/some/path
    systemEventsHookPassword:
      fromVault:
        key: PASSWORD
        path: secret/data/some/path
    internalAPIUser:
      fromVault:
        key: USER
        path: secret/data/some/path
    internalAPIPassword:
      fromVault:
        key: PASSWORD
        path: secret/data/some/path
  listener:
    endpoint:
      dns:
        - backend.example.com
    config:
      redisAsync: false
    marin3r:
      ports:
        - name: backend-http
          port: 38080
        - name: http-internal
          port: 38081
        - name: backend-htttps
          port: 38443
        - name: envoy-metrics
          port: 9901
  worker:
    config:
      redisAsync: true



$ oc apply -f backend.yaml 
backend.saas.3scale.net/example created

Then, the local saas-operator created the grafana dashboard object. I manually delete it:

$ oc delete grafanadashboard backend 

And the error appears.

Now that we have prototyped all controllers using an ansible based operator, we need to move to a go based operator. Main reason behind this is a greater deal of control of the things that need to happen during a deploy. Specifically we need to be able to perform database migrations for System deployments and also manage the trigger of config reloads upon config changes. At least those two are important for the migration of production to OCP.
We will be using https://github.com/redhat-cop/operator-utils which provides a generalized reconciler that can enforce a set of resources, which is basically what we need in every controller, adding extra steps where required.

Why?

We need to add high availability to all workloads managed by saas-operator, so we can ensure that under different circumstances, we can easily escale, and mantain a minimum number of pods up and running, distributed on different hosts and AWS AZs.

How?

When talking about HA, we mean:

• Adding HorizontalPodAutoscaler to deployments that can autoscale (to adapt to incoming traffic)
• Adding PodDisruptionBudget to deployments with more than one pod (so on possible maintenance, we guarantee a minimum or maximum number of pods
• Adding Affinity configuration, so we can guarantee that pods are well distributed among different nodes, of even different AWS AZs

Initially it will be added HA only to backend (backend-listener and backend-worker deployment), after having done some real test, the same configurations will be added to the rest of workloads that can benefit from it (for example, deployments like backend-cron, with a single pod, does not need HA, as a single pod can be running each time)

Requirements

• Add HorizontalPodAutoscaler to backend-worker and backend-listener
• Add PodDisruptionBudget to backend-worker and backend-listener
• Add Affinity configuration to backend-worker and backend-listener

Acceptance Criteria

N/A

Notes

N/A

We have seen, that with current redhat-cop/operator-utils:v1.1.3 (based on operator-sdk v1.3, the same operatorsdk version used at saas-operator), when removing resources like PDB/HPA, randomly they are not deleted and need to be deleted manually (they are not recreated because the controller no longers reconciles it).

It seems the recocile controller to delete it fails, and on next reconcile thinks it is no longer required to watch it.

I have done some changes on backend and system controllers, deploying initially a basic CR:

apiVersion: saas.3scale.net/v1alpha1
kind: Backend
metadata:
  name: example
spec:
  image:
    tag: v3.2.0
  config:
    rackEnv: dev
    redisStorageDSN: backend-redis-storage
    redisQueuesDSN: backend-redis-queues
    systemEventsHookURL:
      fromVault:
        key: URL
        path: secret/data/some/path
    systemEventsHookPassword:
      fromVault:
        key: PASSWORD
        path: secret/data/some/path
    internalAPIUser:
      fromVault:
        key: USER
        path: secret/data/some/path
    internalAPIPassword:
      fromVault:
        key: PASSWORD
        path: secret/data/some/path
  listener:
    loadBalancer:
      eipAllocations:
        - eip-123
        - eip-456
    endpoint:
      dns:
        - backend.example.com
    config:
      redisAsync: false
    marin3r:
      ports:
        - name: backend-http
          port: 38080
        - name: http-internal
          port: 38081
        - name: backend-htttps
          port: 38443
        - name: envoy-metrics
          port: 9901
  worker:
    config:
      redisAsync: true

By default, if not specified, it creates a PDB/HPA for worker and listener:

$ oc get hpa
NAME               REFERENCE                     TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
backend-listener   Deployment/backend-listener   <unknown>/90%   2         4         2          2m10s
backend-worker     Deployment/backend-worker     <unknown>/90%   2         4         2          2m9s

$ oc get pdb
NAME               MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
backend-listener   N/A             1                 0                     2m15s
backend-worker     N/A             1                 0                     2m15s

Then deploy the same CR, but removing their PDB/HPA:

apiVersion: saas.3scale.net/v1alpha1
kind: Backend
metadata:
  name: example
spec:
  image:
    tag: v3.2.0
  config:
    rackEnv: dev
    redisStorageDSN: backend-redis-storage
    redisQueuesDSN: backend-redis-queues
    systemEventsHookURL:
      fromVault:
        key: URL
        path: secret/data/some/path
    systemEventsHookPassword:
      fromVault:
        key: PASSWORD
        path: secret/data/some/path
    internalAPIUser:
      fromVault:
        key: USER
        path: secret/data/some/path
    internalAPIPassword:
      fromVault:
        key: PASSWORD
        path: secret/data/some/path
  listener:
    hpa: {}
    pdb: {}
    loadBalancer:
      eipAllocations:
        - eip-123
        - eip-456
    endpoint:
      dns:
        - backend.example.com
    config:
      redisAsync: false
    marin3r:
      ports:
        - name: backend-http
          port: 38080
        - name: http-internal
          port: 38081
        - name: backend-htttps
          port: 38443
        - name: envoy-metrics
          port: 9901
  worker:
    hpa: {}
    pdb: {}
    config:
      redisAsync: true

And randomly one PDB and HPA (backend or listener), persists and is not deleted, and need to be manually deleted:

$ oc get hpa
NAME             REFERENCE                   TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
backend-worker   Deployment/backend-worker   <unknown>/90%   2         4         2          15s

$ oc get pdb
NAME             MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
backend-worker   N/A             1                 0                     16s
slopez @ ~/work/sidekiq-split/saas-operator

After doing some tests using newer redhat-cop/operator-utils:v1.1.4 (based on operator-sdk v1.9), the issue seems to be fixed, at least I have not been able to reproduce it, so when removing PDB/HPA from CR spec, they are really deleted.

Why?

We need to add more 3scale components to saas-operator as new operator controller (ansible role). CORSProxy entire stack has been deployed in staging, and is ready to be encapsulated in the operator.

How?

Add a new corsproxy controller with operator-sdk binary, extracting all yamls used on staging environment: deployment, service, podmonitor, secretdefinition, ingress, and grafana dashboard.

By the moment prometheusrules will be out of the operator scope, as they can be very dynamic, still figuring out best alerts, thresholds...

Requirements

• Add corsproxy controller with all required objects in a single CRD

Acceptance Criteria

N/A

Notes

N/A

container_memory_usage_bytes includes cached items that can be evicted under memory pressure. The better metric is container_memory_working_set_bytes as this is what the OOM killer is watching for.

Grafana dashboards are using the container_memory_usage_bytes for memory usage / quota panels and should be updated.

Right now we have a couple of system tasks defined https://github.com/3scale/3scale-saas/tree/f0d96a5ee51b2a2a537f57d8ad09e62d98d76900/manifests/stg-saas/ocp4/3scale-saas/tasks that are executed manually on demmand:

• system-backend-sync
• system-db-migrate

They are embedded into k8s job and a manual ExternalSecret, and we need to check what to do with them, because managing them manually does not scale as long as saas-operator already knows the required system secrets.

Either integratig them into saas-operator, or maybe just document them to be executed from the system ruby console statefulset.

/assign @raelga
/kind feature
/priority important-soon
/label size/l

Closes https://github.com/3scale/3scale-saas/issues/144

name: Bug Report
about: Report a bug encountered while using the module
title: 'bug: deployment replica count reconciliation'
labels: kind/bug

What happened:

When any saas-operator resource instance spec is updated, the workers try to apply the stored state into the Kubernetes resources owned by the operator. For deployments, this causes that the replica count to be defaulted. When using HPA, this default triggers a scale down to 1 (or the default) and then a scale up (to the current HPA replica count).

What you expected to happen:

The deployment replica count when updating a saas-operator resource is retrieved from the current state instead of defaulted.

How to reproduce it (as minimally and precisely as possible):

1. Deploy any saas-operator resource instance
2. Enable HPA with at least 2 minReplicas
3. Update the saas-operator resource instance (any attribute)

At this point, the deployment will be updated, scaling down to 1 pod and terminating the rest of the pods. Few seconds after, HPA reconciler will kick in and update the Deployment replica count back to the value before the resource update.

Environment:

• Kubernetes version: v1.19.0
• OpenShift version: 4.6.19
• Operator version: 0.9.5

Why?

We need to add more 3scale components to saas-operator as new operator controller (ansible role). EchoAPI entire stack has been deployed in staging, and is ready to be encapsulated in the operator.

How?

Add a new echoapi controller with operator-sdk binary, extracting all yamls used on staging environment: deployment, service and marin3r envoyconfig

Requirements

• Add echoapi controller with all required objects in a single CRD

Acceptance Criteria

N/A

Notes

N/A

On dev environment (OCP 4.8), there is the following warning:

W0920 17:50:17.379186       1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget

On OCP 4.7 (staging/production), we are using v1beta1 because is the only version available.

Once we update staging/production to OCP 4.8, we should update PDB API to v1

There are some configuration blocks that we might not need for SaaS. We should check with system team and remove them in case we get confirmation they are not necessary (or at least make them optional).

Here is a custom resource with comments on the fields that could be removed/changed:

apiVersion: saas.3scale.net/v1alpha1
kind: System
metadata:
  name: example
spec:
  image:
    pullPolicy: Always
  config:
    # TODO: can be removed?
    seed:
      masterAccessToken:
        fromVault:
          path: secret/data/path/system-seed
          key: MASTER_ACCESS_TOKEN
      masterDomain: multitenant-admin
      masterUser:
        fromVault:
          path: secret/data/path/system-seed
          key: MASTER_USER
      masterPassword:
        fromVault:
          path: secret/data/path/system-seed
          key: MASTER_PASSWORD
      adminAccessToken:
        fromVault:
          path: secret/data/path/system-seed
          key: ADMIN_ACCESS_TOKEN
      adminUser:
        fromVault:
          path: secret/data/path/system-seed
          key: ADMIN_USER
      adminPassword:
        fromVault:
          path: secret/data/path/system-seed
          key: ADMIN_PASSWORD
      adminEmail: [email protected]
      tenantName: example
    databaseDSN:
      fromVault:
        path: secret/data/path/system-database
        key: URL
    databaseSecret:
      fromVault:
        path: secret/data/path/system-app
        key: DB_SECRET
    eventsSharedSecret:
      fromVault:
        path: secret/data/path/system-app
        key: EVENTS_SHARED_SECRET
    secretKeyBase:
      fromVault:
        path: secret/data/path/system-app
        key: SECRET_KEY_BASE
    recaptcha:
      publicKey:
        fromVault:
          path: secret/data/path/system-recaptcha
          key: RECAPTCHA_PUBLIC_KEY
      privateKey:
        fromVault:
          path: secret/data/path/system-recaptcha
          key: RECAPTCHA_PRIVATE_KEY
    accessCode:
      fromVault:
        path: secret/data/path/system-app
        key: ACCESS_CODE
    # TODO: is this used?
    segment:
      deletionWorkspace: example
      deletionToken:
        fromVault:
          path: secret/data/path/system-app
          key: SEGMENT_DELETION_TOKEN
      writeKey:
        fromVault:
          path: secret/data/path/system-app
          key: SEGMENT_WRITE_KEY
    github:
      clientID:
        fromVault:
          path: secret/data/path/system-app
          key: GITHUB_CLIENT_ID
      clientSecret:
        fromVault:
          path: secret/data/path/system-app
          key: GITHUB_CLIENT_SECRET
    # TODO: can be removed?
    metrics:
      user:
        fromVault:
          path: secret/data/path/system-app
          key: PROMETHEUS_USER
      password:
        fromVault:
          path: secret/data/path/system-app
          key: PROMETHEUS_PASSWORD
    redhatCustomerPortal:
      clientID:
        fromVault:
          path: secret/data/path/system-app
          key: RH_CUSTOMER_PORTAL_CLIENT_ID
      clientSecret:
        fromVault:
          path: secret/data/path/system-app
          key: RH_CUSTOMER_PORTAL_CLIENT_SECRET
    bugsnag:
      apiKey:
        fromVault:
          path: secret/data/path/system-app
          key: BUGSNAG_API_KEY
    memcachedServers: ""
    redis:
      queuesDSN: redis://system-redis:6379/1
      messageBusDSN: redis://system-redis:6379/2
    smtp:
      address: localhost
      user:
        fromVault:
          path: secret/data/path/system-smtp
          key: username
      password:
        fromVault:
          path: secret/data/path/system-smtp
          key: password
      port: 25
      authProtocol: none
      opensslVerifyMode: none
      starttlsAuto: false
    apicastAccessToken:
      fromVault:
        path: secret/data/path/system-master-apicast
        key: ACCESS_TOKEN
    zyncAuthToken:
      fromVault:
        path: secret/data/path/zync
        key: ZYNC_AUTHENTICATION_TOKEN
    backend:
      externalEndpoint: https://backend.example.com
      internalEndpoint: http://backend-listener-internal
      redisDSN: redis://backend-redis:6379/0
      internalAPIUser:
        fromVault:
          path: secret/data/path/backend-internal-api
          key: username
      internalAPIPassword:
        fromVault:
          path: secret/data/path/backend-internal-api
          key: password
    assets:
      # TODO: cdn url??
      bucket: my-bucket
      region: us-east-1
      accessKey:
        fromVault:
          path: secret/data/path/system-multitenant-assets-s3
          key: AWS_ACCESS_KEY_ID
      secretKey:
        fromVault:
          path: secret/data/path/system-multitenant-assets-s3
          key: AWS_SECRET_ACCESS_KEY