3scale-ops / saas-operator Goto Github PK
View Code? Open in Web Editor NEW3scale SaaS Operator - www.3scale.net
License: Apache License 2.0
3scale SaaS Operator - www.3scale.net
License: Apache License 2.0
With #144, a new workload reconciler has been implemented and autossl
is a good fit for it.
/kind feature
/priority important-longterm
/assign
Add a new apicast controller with operator-sdk
binary, extracting all yamls used on staging environment.
N/A
N/A
EchoAPI was the first workload to have marin3r managed with saas-operator. Lately with Backend and Apicast, we changed the way we manage marin3r annotations thrpugh the CR, so these changes need to be applied also to EchoAPI controller, so all workloads use the same framework to configure similar things.
Apply Backend marin3r config from #19 to EchoAPI controller
N/A
N/A
With #144, a new workload reconciler has been implemented and system
is a good fit for it.
/kind feature
/priority important-longterm
/assign
Information of zync-que jobs is not showing the data per job type (showing Value
), because the used label as grafana legend is label exported_job
which don't exist:
Actually, this is weird, normally having a label exported_whatever
is caused because using a reserved prometheus label into app metric labels (which is wrong from the app point of view), so when prometheus scrape the app, adds the preffix exported_X
and replace the original label, to differenciate from label job (prometheus target job).
It seems it has ben renamed to job_name
at PR 3scale/zync#283 (it is needed to be checked other possible changes that affects the dashboard)
Move quay.io/3scaleops/saas-operator-bundle
to quay.io/3scale
. Release workflow will required modifications.
Following issue #55, now is the moment to convert current backend controller from ansible to go, following the new reconciler provided by https://github.com/redhat-cop/operator-utils , which has been already used for Apicast, MappingService and AutoSSL controllers at #57
We need to add more 3scale components to saas-operator as new operator controller (ansible role). Zync entire stack has been deployed in staging, and is ready to be encapsulated in the operator.
Add a new zync controller with operator-sdk
binary, extracting all yamls used on staging environment: deployments, services, podmonitors, secretdefinitions, and grafana dashboards.
By the moment prometheusrules will be out of the operator scope, as they can be very dynamic, still figuring out best alerts, thresholds...
N/A
OperatorSDK v0.x has been deprecated and superseded by v.1.x that introduces several breaking changes. As part of the OLM publishing #42, this issue will tackle the migration to v1.x prior to the OLM bundling.
Implement a mechanism, usable by any controller, that causes Deployment rollouts on Secret or ConfigMap changes.
The implementation will be:
It will be first implemented/tested for MappingService controller, which uses a Secret as a source for an environment variable.
After deploying system onto OCP, several improvements to the 3scale SaaS Operator had been discussed.
By adding all the required functionalities to the system workload (and the others if required).
/kind feature
/priority important-soon
/assign
We need to add more 3scale components to saas-operator as new operator controller (ansible role). System entire stack has been deployed in staging, and is ready to be encapsulated in the operator.
Add a new System controller with operator-sdk
binary, extracting all yamls used on staging environment: deployments, services, podmonitors, secretdefinitions, and grafana dashboards.
N/A
System is the only controller from saas-operator who don't include a dashboard.
Use the product dashboard from 3scale-operator as a source to implement the system saas-operator.
Soon we are going to migrate current boxy layer in EC2 (sentinel + twemproxy) to OCP, and we need to have good serviceability around it:
Both sentinel and twemproxy has already dashboards and alerts scrapped from EC2 by the prometheus-ec2 deployment, but now we need to convert what we have to be deployed by operator, doing lot of required changes due to the difference in the way are deployed
So it is needed to:
Review and list differences between the 3scale/saas-operator requirements and the actual 3scale/3scale-operator.
While working on https://github.com/3scale/infrastructure/pull/1011 I have seen that there are a lot of backen-listener internal metrics that are missing on backend dashboard from saas-operator, but appears on the official operator 3scale/3scale-operator#473
So it is needed to add them
With #144, a new workload reconciler has been implemented and cors-proxy
is a good fit for it.
In addition, with #150, SecretDefinition can be now replaced with ExternalSecret, so it will be the first workload using the new External Secrets Operator (to be deployed in production clusters yet).
/kind feature
/priority important-longterm
/assign
At the moment we are using temporarily a OpenShift Route
to publish Backend listener service. Now it is the moment to remove it, and publish the service with a AWS NLB (attaching an EIP created with TF via service annotation), and add marin3r envoy sidecar in front of backend-listener deployment, doing TLS termination, and doing ratelimit at infra level (by the moment only by IP, X-forwarded-proto header).
Add maybe a new backend-listener-gateway service (or replace current one) of type LoadBalancer
, AWS NLB annotation, annotations to configure cross zone load balancing, annottion to configure proxy protocol enabled via aws-nlb-helper-operator , and configure marin3r status label and free-form annotations on backend listener deployment (everything like (like echoapi).
N/A
N/A
Until now, we had separated twemproxy instances for Backend and System.
However, upon a master failover, slaves are not ready during a few minutes, and so for a while System needs to contact twemproxy-master to make possible contact with Backend redis (not ideal, but not bad for a few minutes during a failover which happens rarely).
In the legacy platform, the twemproxy-slave-rw EC2 instances had a custom metric reporting that situation, so although being a slave-rw instance, they have configured a master or a slave-readonly-yes. These are the associated prometheus alerts:
- alert: BackendStorageBoxySlaveWithMaster
expr: boxy_is_slave{boxy_role="slave"} < 1
for: 5m
labels:
severity: critical
annotations:
message: "Backend redis storage instance {{ $labels.slave }} in {{ $labels.instance }} it's a master. Servers on {{ $labels.instance}} should be slaves, not masters"
- alert: BackendStorageBoxySlaveWithSlaveReadOnly
expr: boxy_read_only_slave{boxy_role="slave"} > 0
for: 5m
labels:
severity: critical
annotations:
message: "Backend redis storage instance {{ $labels.slave }} in {{ $labels.instance }} it's a slave read_only=yes. Servers on {{ $labels.instance }} should be slaves with read_only=no"
So, in case of these alerts firing, oncall is paged telling us that this twemproxy-slave-rw used by System is missconfigured and need proper manual configuration, telling us with labels which is the wrong associated shard and the wrong associated redis-server configured.
However, right now with the new approach of TwemproxyConfig and twemproxy sidecars, we have lost this alert.
In the peace of code at TwemproxyConfig generator
saas-operator/pkg/generators/twemproxyconfig/generator.go
Lines 157 to 163 in 55d0213
In case of having a twemproxy-slaves-rw TwemproxyConfig, if there are no available slaves-rw, twemproxy config will point to master. On next reconcile period (every 30 seconds), it will be checked again, and once slaves-rw gets available again (which normally require manual human intervention to update roles, so can take a few minutes to happen), twemproxy-slave-rw will do a fallback (will stop traffic to master and point to slaves-rw again).
However, there is no prometheus metric telling us we are in the state of a twemproxy-slave-rw using masters, which is very useful, because upon a failover, we normally need to manually update redis roles (1 master, 1 slave-rw-no, and 1 slave-rw-yes), and can take between 15 and 30 minutes (if we forgot about it, thanks to the alert we know it needs to be done)
To not lose an important metric we have in the legacy twemproxy environment, we should have a counter metric reporting that a twemproxy-slave-rw is using a master, so that way we can add an alert with a ratefunction that will page oncall in case it happens for more than X minutes.
- alert: TwemproxySlave
expr: rate(saas_twemproxy_slave_rw_using_master[1m]) > 0
for: 1m
labels:
severity: critical
annotations:
message: "Twemproxy slave is pointing to a redis master instead a slave-rw!!!"
Having System pointing to master is dangerous, from time to time System does redis slow queries of more than a minute doing sorts, which blocks the whole redis shard, that why System ideally should always point to slaves-rw.
The system-sidekiq-default with current value:
-q critical -q backend_sync -q events -q zync,40 -q priority,25 -q default,15 -q web_hooks,10 -q deletion,5
Is not loading correctly the argument by System-sidekiq (note the Queues field):
Because it should be an array of strings separated by space, we can use something like strings.Split()
.
In addition, the go code refer to it as QueuesArg
(singular), while the CRD field is called queuesArgs
(plural), as the CR field is not yet used on any CR, we should use either one or the other, so we might just update the CR field to queuesArg
.
Is your feature request related to a problem? Please describe.
We want to publish saas-operator on OperatorHub.io through OLM, so anyone can easily deploy it.
Describe the solution you'd like
Follow https://operator-framework.github.io/olm-book/ instructions to make this operator OLM "compliant" and make a pull request to include in the Community operators marketplace: https://github.com/operator-framework/community-operators/blob/master/docs/contributing.md
Describe alternatives you've considered
Manual installation using all the Kubernetes objects required by the operator (deployment, rbac, ...).
Additional context
Good example of operator contribution: operator-framework/community-operators#1934
With #144, a new workload reconciler has been implemented and echo-api
is a good fit for it.
/kind feature
/priority important-longterm
/assign
Operators everywhere ...
Add a new mapping-service controller with operator-sdk
binary, extracting all yamls used on staging environment.
N/A
N/A
While working on sentinel grafana dashboard #197 I discovered a couple of bugs:
All standard metrics are retrieved from sentinel commands.
For every redis-server on a given shard with a given role, there is a given timeseries database (tuple with 3 elements)
However, when there is any change of the role from a given redis-server, it is a created a new timeseriesDB for the new shard--role2--redis-server, however the old shard--role1--redis-server timeseriesDB keeps being reported with latest value, although this tuple does not exist anymore.
This redis server 10.65.6.5
from shard01
was a master long time ago (in yellow) but it is now a slave (blue)
And this apply to any metric where the role label is added in https://github.com/3scale-ops/saas-operator/blob/8aafa688780f86dcb013bf8bf7fe884e1bf44d43/pkg/redis/metrics/sentinel_metrics.go
A workaround would be to remove the role label from every metric, so the timeseries would incldue the tuple of shard-redis-server (instead of shard--role--redis-server), however I think being sentinel metrics it make sense to always have this role label.
IMO, the ideal solution would be to stop reporting metrics whose tuple shard--role--redis-server in not active anymore.
master
to slave
role, what we would should see is:
master
and now is a slave
High Availability configuration (HorizontalPodAutoscaler
, PodDisruptionBudget
, podAntiaffinity
) was added to Backend controller on ISSUE #8 (PR #29).
Now it is the moment to add the same features to the rest of the components.
Repeat what it was done on #29 regarding HA (HorizontalPodAutoscaler
, PodDisruptionBudget
, podAntiaffinity
):
To the rest of controllers:
It needs to be added to all deployment that can 3scale (for example, backend-cron just needs a single pod, so it is not needed), the same applies for example to system-sphinx.
HorizontalPodAutoscaler
, PodDisruptionBudget
, podAntiaffinity
) to ApicastHorizontalPodAutoscaler
, PodDisruptionBudget
, podAntiaffinity
) to ZyncHorizontalPodAutoscaler
, PodDisruptionBudget
, podAntiaffinity
) to AutoSSLHorizontalPodAutoscaler
, PodDisruptionBudget
, podAntiaffinity
) to CORSProxyHorizontalPodAutoscaler
, PodDisruptionBudget
, podAntiaffinity
) to MappingServiceHorizontalPodAutoscaler
, PodDisruptionBudget
, podAntiaffinity
) to EchoAPIN/A
N/A
We are in the process of switching from SecretDefinitons
(from secrets-manager) to ExternalSecrets
(external-secrets operator).
We need to update our saas-operator to create this new ExternalSecrets
custom resources instead of current SecretDefinitons
.
We will need to update current CRDs by adding to the fromVault
spec:
vault-mgmt
, so no need to update current CRs with this new field)60s
, so no need to update current CRs with this new field)secrets/data/
if set (not needed on ExternalSecrets), so no need to update current CRs with this existing fieldExample, from:
apiVersion: secrets-manager.tuenti.io/v1alpha1
kind: SecretDefinition
metadata:
name: system-database-seed
spec:
name: system-database-seed
type: Opaque
keysMap:
DB_NAME:
path: secret/data/kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
key: DB_NAME
DB_USER:
path: secret/data/kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
key: DB_USER
DB_PASSWORD:
path: secret/data/kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
key: DB_PASSWORD
URL:
path: secret/data/kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
key: URL
To:
apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
name: system-database-seed
spec:
refreshInterval: 60s
secretStoreRef:
name: vault-mgmt
kind: ClusterSecretStore
target:
name: system-database-seed
data:
- secretKey: DB_NAME
remoteRef:
key: kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
property: DB_NAME
- secretKey: DB_USER
remoteRef:
key: kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
property: DB_USER
- secretKey: DB_PASSWORD
remoteRef:
key: kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
property: DB_PASSWORD
- secretKey: URL
remoteRef:
key: kubernetes/dev-eng/ocp4-5/3scale-saas/system-database
property: URL
As the target secrets will remain the same (the secret.spec.data
won't change, so won't be triggered any deployment), to avoid the target secrets being removed temporarily and recreated by external-secrets-operator, it will be needed to:
ExternalSecret
will be created by saas-operator, and will adopt current target secret without doing changes on the secrets data (if it is configured correctly), which is the one used by saas-operator hashes:
ExternalSecret
resource which creates a Secret
resourceExternalSecrets
propagated labels and current ones from secrets-manager (last updated date
, and updatedBy secrets-manager
)SecretDefinition
won't be deleted, you need to manually edit each SecretDefinition
by removing the metadata.finalizers
field, then the SecretDefinition
can be deletedlast updated date
and updatedBy secrets-manager
from target secretsWe need to add more 3scale components to saas-operator as new operator controller (ansible role). One of the main and more complex components (in terms of number of objects involved...) is 3scale backend, so it is the next controller to be implemented, reducing the complexity of application management to a single and small CR (to handle differences between environements with CR fields...).
Add a new backend controller with operator-sdk
binary, extracting all yamls used on staging environment: deployments, services, route, configmap, podmonitors, secretdefinitions, and grafana dashboards (backned main app dashboard, and all dashboards related to storage on EC2, nodes, redis, sentinel, twemproxy...).
By the moment prometheusrules will be out of the operator scope, as they can be very dynamic, still figuring out best alerts, thresholds... (so there is no need to update operator image with a new release to add, finetune, delete alerts or apply small changes).
N/A
N/A
While working on sentinel grafana dashboard #197 I discovered a couple of bugs:
Most of the sentinel metrics are retrieved from a few sentinel commands, and are gauges.
However there are 3 counter metrics which are the most important ones defined at https://github.com/3scale-ops/saas-operator/blob/8aafa688780f86dcb013bf8bf7fe884e1bf44d43/pkg/redis/events/watcher.go:
The thing is, being a counter, we need to use prometheus rate function, to see increases over time (1 minute for example):
rate(saas_redis_sentinel_sdown_count{}[1m])
So something like (indicating 2 consecutive failovers on a given shard):
However:
1
, which means that rate function is 0
:So:
0
to 1
, and can be used on both dashboard and alertsOn e1ea6e6 it was update sentinel SubjectiveDown metric label, from server
to redis_server
, to use same convention as all the other sentinel metrics.
Don't know if it was introduced a bug, or that directly rem.target.ip
is the sentinel pod ip (range 10.130.X.X). while we want the label redis_server
to be filled with each monitored redis instance IP (range 10.65.X.X).
This is causing, that together with a descheduler problem (restarting sentinel pods with constant evictions), to report SDOWN alerts costantly because sentinel pods are recreated with diferent IPs (but of course using same hostname as it is an statefulset).
While working on some grafana dashboards improvements, I removed manually a grafana dashboard managed by my local saas-operator, and the dashboard was not being recreated, reporting error "unable to get: /slopez because of unknown namespace for the cache"
(slopez
was my namespace):
2021-03-02T10:48:56.654+0100 ERROR resource-reconciler.slopez/example.integreatly.org/v1alpha1/GrafanaDashboard/slopez/backend unable to retrieve {"namespace": "e.Meta.GetNamespace()", "error": "unable to get: /slopez because of unknown namespace for the cache"}
github.com/go-logr/zapr.(*zapLogger).Error
/home/slopez/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*resourceModifiedPredicate).Delete
/home/slopez/go/pkg/mod/github.com/redhat-cop/operator-utils@v1.1.1/pkg/util/lockedresourcecontroller/resource-reconciler.go:213
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
/home/slopez/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/source/internal/eventsource.go:131
k8s.io/client-go/tools/cache.(*processorListener).run.func1
/home/slopez/go/pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/shared_informer.go:779
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/home/slopez/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/home/slopez/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/home/slopez/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
/home/slopez/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run
/home/slopez/go/pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/shared_informer.go:771
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
/home/slopez/go/pkg/mod/k8s.io/apimachinery@v0.20.0/pkg/util/wait/wait.go:73
I executed the operator locally watching a specific namespace:
make run WATCH_NAMESPACE=slopez
On a different terminal, I created a sample Backend
CR on the watched namespace:
$ cat backend.yaml
apiVersion: saas.3scale.net/v1alpha1
kind: Backend
metadata:
name: example
namespace: slopez
spec:
image:
tag: v3.2.0
config:
rackEnv: dev
redisStorageDSN: backend-redis-storage
redisQueuesDSN: backend-redis-queues
systemEventsHookURL:
fromVault:
key: URL
path: secret/data/some/path
systemEventsHookPassword:
fromVault:
key: PASSWORD
path: secret/data/some/path
internalAPIUser:
fromVault:
key: USER
path: secret/data/some/path
internalAPIPassword:
fromVault:
key: PASSWORD
path: secret/data/some/path
listener:
endpoint:
dns:
- backend.example.com
config:
redisAsync: false
marin3r:
ports:
- name: backend-http
port: 38080
- name: http-internal
port: 38081
- name: backend-htttps
port: 38443
- name: envoy-metrics
port: 9901
worker:
config:
redisAsync: true
$ oc apply -f backend.yaml
backend.saas.3scale.net/example created
Then, the local saas-operator created the grafana dashboard object. I manually delete it:
$ oc delete grafanadashboard backend
And the error appears.
Now that we have prototyped all controllers using an ansible based operator, we need to move to a go based operator. Main reason behind this is a greater deal of control of the things that need to happen during a deploy. Specifically we need to be able to perform database migrations for System deployments and also manage the trigger of config reloads upon config changes. At least those two are important for the migration of production to OCP.
We will be using https://github.com/redhat-cop/operator-utils which provides a generalized reconciler that can enforce a set of resources, which is basically what we need in every controller, adding extra steps where required.
We need to add high availability to all workloads managed by saas-operator, so we can ensure that under different circumstances, we can easily escale, and mantain a minimum number of pods up and running, distributed on different hosts and AWS AZs.
When talking about HA, we mean:
HorizontalPodAutoscaler
to deployments that can autoscale (to adapt to incoming traffic)PodDisruptionBudget
to deployments with more than one pod (so on possible maintenance, we guarantee a minimum or maximum number of podsAffinity configuration
, so we can guarantee that pods are well distributed among different nodes, of even different AWS AZsInitially it will be added HA only to backend (backend-listener and backend-worker deployment), after having done some real test, the same configurations will be added to the rest of workloads that can benefit from it (for example, deployments like backend-cron, with a single pod, does not need HA, as a single pod can be running each time)
HorizontalPodAutoscaler
to backend-worker and backend-listenerPodDisruptionBudget
to backend-worker and backend-listenerAffinity configuration
to backend-worker and backend-listenerN/A
N/A
We have seen, that with current redhat-cop/operator-utils:v1.1.3
(based on operator-sdk v1.3
, the same operatorsdk version used at saas-operator), when removing resources like PDB/HPA, randomly they are not deleted and need to be deleted manually (they are not recreated because the controller no longers reconciles it).
It seems the recocile controller to delete it fails, and on next reconcile thinks it is no longer required to watch it.
I have done some changes on backend and system controllers, deploying initially a basic CR:
apiVersion: saas.3scale.net/v1alpha1
kind: Backend
metadata:
name: example
spec:
image:
tag: v3.2.0
config:
rackEnv: dev
redisStorageDSN: backend-redis-storage
redisQueuesDSN: backend-redis-queues
systemEventsHookURL:
fromVault:
key: URL
path: secret/data/some/path
systemEventsHookPassword:
fromVault:
key: PASSWORD
path: secret/data/some/path
internalAPIUser:
fromVault:
key: USER
path: secret/data/some/path
internalAPIPassword:
fromVault:
key: PASSWORD
path: secret/data/some/path
listener:
loadBalancer:
eipAllocations:
- eip-123
- eip-456
endpoint:
dns:
- backend.example.com
config:
redisAsync: false
marin3r:
ports:
- name: backend-http
port: 38080
- name: http-internal
port: 38081
- name: backend-htttps
port: 38443
- name: envoy-metrics
port: 9901
worker:
config:
redisAsync: true
By default, if not specified, it creates a PDB/HPA for worker and listener:
$ oc get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
backend-listener Deployment/backend-listener <unknown>/90% 2 4 2 2m10s
backend-worker Deployment/backend-worker <unknown>/90% 2 4 2 2m9s
$ oc get pdb
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
backend-listener N/A 1 0 2m15s
backend-worker N/A 1 0 2m15s
Then deploy the same CR, but removing their PDB/HPA:
apiVersion: saas.3scale.net/v1alpha1
kind: Backend
metadata:
name: example
spec:
image:
tag: v3.2.0
config:
rackEnv: dev
redisStorageDSN: backend-redis-storage
redisQueuesDSN: backend-redis-queues
systemEventsHookURL:
fromVault:
key: URL
path: secret/data/some/path
systemEventsHookPassword:
fromVault:
key: PASSWORD
path: secret/data/some/path
internalAPIUser:
fromVault:
key: USER
path: secret/data/some/path
internalAPIPassword:
fromVault:
key: PASSWORD
path: secret/data/some/path
listener:
hpa: {}
pdb: {}
loadBalancer:
eipAllocations:
- eip-123
- eip-456
endpoint:
dns:
- backend.example.com
config:
redisAsync: false
marin3r:
ports:
- name: backend-http
port: 38080
- name: http-internal
port: 38081
- name: backend-htttps
port: 38443
- name: envoy-metrics
port: 9901
worker:
hpa: {}
pdb: {}
config:
redisAsync: true
And randomly one PDB and HPA (backend or listener), persists and is not deleted, and need to be manually deleted:
$ oc get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
backend-worker Deployment/backend-worker <unknown>/90% 2 4 2 15s
$ oc get pdb
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
backend-worker N/A 1 0 16s
slopez @ ~/work/sidekiq-split/saas-operator
After doing some tests using newer redhat-cop/operator-utils:v1.1.4
(based on operator-sdk v1.9
), the issue seems to be fixed, at least I have not been able to reproduce it, so when removing PDB/HPA from CR spec, they are really deleted.
We need to add more 3scale components to saas-operator as new operator controller (ansible role). CORSProxy entire stack has been deployed in staging, and is ready to be encapsulated in the operator.
Add a new corsproxy controller with operator-sdk
binary, extracting all yamls used on staging environment: deployment, service, podmonitor, secretdefinition, ingress, and grafana dashboard.
By the moment prometheusrules will be out of the operator scope, as they can be very dynamic, still figuring out best alerts, thresholds...
N/A
N/A
container_memory_usage_bytes
includes cached items that can be evicted under memory pressure. The better metric is container_memory_working_set_bytes
as this is what the OOM killer is watching for.
Grafana dashboards are using the container_memory_usage_bytes
for memory usage / quota panels and should be updated.
Right now we have a couple of system tasks defined https://github.com/3scale/3scale-saas/tree/f0d96a5ee51b2a2a537f57d8ad09e62d98d76900/manifests/stg-saas/ocp4/3scale-saas/tasks that are executed manually on demmand:
They are embedded into k8s job and a manual ExternalSecret
, and we need to check what to do with them, because managing them manually does not scale as long as saas-operator already knows the required system secrets.
Either integratig them into saas-operator, or maybe just document them to be executed from the system ruby console statefulset.
/assign @raelga
/kind feature
/priority important-soon
/label size/l
name: Bug Report
about: Report a bug encountered while using the module
title: 'bug: deployment replica count reconciliation'
labels: kind/bug
When any saas-operator resource instance spec is updated, the workers try to apply the stored state into the Kubernetes resources owned by the operator. For deployments, this causes that the replica count to be defaulted. When using HPA, this default triggers a scale down to 1 (or the default) and then a scale up (to the current HPA replica count).
The deployment replica count when updating a saas-operator
resource is retrieved from the current state instead of defaulted.
saas-operator
resource instanceminReplicas
saas-operator
resource instance (any attribute)At this point, the deployment will be updated, scaling down to 1 pod and terminating the rest of the pods. Few seconds after, HPA reconciler will kick in and update the Deployment replica count back to the value before the resource update.
We need to add more 3scale components to saas-operator as new operator controller (ansible role). EchoAPI entire stack has been deployed in staging, and is ready to be encapsulated in the operator.
Add a new echoapi controller with operator-sdk
binary, extracting all yamls used on staging environment: deployment, service and marin3r envoyconfig
N/A
N/A
On dev environment (OCP 4.8), there is the following warning:
W0920 17:50:17.379186 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
On OCP 4.7 (staging/production), we are using v1beta1
because is the only version available.
Once we update staging/production to OCP 4.8, we should update PDB API to v1
There are some configuration blocks that we might not need for SaaS. We should check with system team and remove them in case we get confirmation they are not necessary (or at least make them optional).
Here is a custom resource with comments on the fields that could be removed/changed:
apiVersion: saas.3scale.net/v1alpha1
kind: System
metadata:
name: example
spec:
image:
pullPolicy: Always
config:
# TODO: can be removed?
seed:
masterAccessToken:
fromVault:
path: secret/data/path/system-seed
key: MASTER_ACCESS_TOKEN
masterDomain: multitenant-admin
masterUser:
fromVault:
path: secret/data/path/system-seed
key: MASTER_USER
masterPassword:
fromVault:
path: secret/data/path/system-seed
key: MASTER_PASSWORD
adminAccessToken:
fromVault:
path: secret/data/path/system-seed
key: ADMIN_ACCESS_TOKEN
adminUser:
fromVault:
path: secret/data/path/system-seed
key: ADMIN_USER
adminPassword:
fromVault:
path: secret/data/path/system-seed
key: ADMIN_PASSWORD
adminEmail: [email protected]
tenantName: example
databaseDSN:
fromVault:
path: secret/data/path/system-database
key: URL
databaseSecret:
fromVault:
path: secret/data/path/system-app
key: DB_SECRET
eventsSharedSecret:
fromVault:
path: secret/data/path/system-app
key: EVENTS_SHARED_SECRET
secretKeyBase:
fromVault:
path: secret/data/path/system-app
key: SECRET_KEY_BASE
recaptcha:
publicKey:
fromVault:
path: secret/data/path/system-recaptcha
key: RECAPTCHA_PUBLIC_KEY
privateKey:
fromVault:
path: secret/data/path/system-recaptcha
key: RECAPTCHA_PRIVATE_KEY
accessCode:
fromVault:
path: secret/data/path/system-app
key: ACCESS_CODE
# TODO: is this used?
segment:
deletionWorkspace: example
deletionToken:
fromVault:
path: secret/data/path/system-app
key: SEGMENT_DELETION_TOKEN
writeKey:
fromVault:
path: secret/data/path/system-app
key: SEGMENT_WRITE_KEY
github:
clientID:
fromVault:
path: secret/data/path/system-app
key: GITHUB_CLIENT_ID
clientSecret:
fromVault:
path: secret/data/path/system-app
key: GITHUB_CLIENT_SECRET
# TODO: can be removed?
metrics:
user:
fromVault:
path: secret/data/path/system-app
key: PROMETHEUS_USER
password:
fromVault:
path: secret/data/path/system-app
key: PROMETHEUS_PASSWORD
redhatCustomerPortal:
clientID:
fromVault:
path: secret/data/path/system-app
key: RH_CUSTOMER_PORTAL_CLIENT_ID
clientSecret:
fromVault:
path: secret/data/path/system-app
key: RH_CUSTOMER_PORTAL_CLIENT_SECRET
bugsnag:
apiKey:
fromVault:
path: secret/data/path/system-app
key: BUGSNAG_API_KEY
memcachedServers: ""
redis:
queuesDSN: redis://system-redis:6379/1
messageBusDSN: redis://system-redis:6379/2
smtp:
address: localhost
user:
fromVault:
path: secret/data/path/system-smtp
key: username
password:
fromVault:
path: secret/data/path/system-smtp
key: password
port: 25
authProtocol: none
opensslVerifyMode: none
starttlsAuto: false
apicastAccessToken:
fromVault:
path: secret/data/path/system-master-apicast
key: ACCESS_TOKEN
zyncAuthToken:
fromVault:
path: secret/data/path/zync
key: ZYNC_AUTHENTICATION_TOKEN
backend:
externalEndpoint: https://backend.example.com
internalEndpoint: http://backend-listener-internal
redisDSN: redis://backend-redis:6379/0
internalAPIUser:
fromVault:
path: secret/data/path/backend-internal-api
key: username
internalAPIPassword:
fromVault:
path: secret/data/path/backend-internal-api
key: password
assets:
# TODO: cdn url??
bucket: my-bucket
region: us-east-1
accessKey:
fromVault:
path: secret/data/path/system-multitenant-assets-s3
key: AWS_ACCESS_KEY_ID
secretKey:
fromVault:
path: secret/data/path/system-multitenant-assets-s3
key: AWS_SECRET_ACCESS_KEY
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.