globocom / enforcement

Project focused on the implementation of policies in Kubernetes clusters through GitOps.

Enforcement

Introduction

Enforcement is an open source project focused on the management and simultaneous deployment of applications and policies across multiple clusters through GitOps.

Enforcement allows users to manage many clusters as easily as one. Users can deploy packages (resource collection) to clusters created from a cluster source (Rancher, GKE, EKS, etc.) and control deployments by specifying rules, which define a filter to select a group of clusters and the packages that should be installed in that group.

When Enforcement detects the creation of a cluster in the cluster source, it checks whether the cluster fits into any specified rule, if there is any match, the packages configured in the rule are automatically installed in the cluster.

The packages include not just application deployment manifests, but anything that can be described as a feature of Kubernetes.

How does it work?

Enforcement works as a Kubernetes Operator, which observes the creation of ClusterRule objects. These objects define rules that specify a set of clusters and the packages they are to receive.

When Enforcement detects the creation of a cluster in cluster source, it registers the cluster and asks ArgoCD to install the packages configured for the cluster.

ArgoCD installs all packages in the cluster and ensures that they are always present.

Installation

Enforcement can be installed on Kubernetes using a helm chart. See the following page for information on how to get Enforcement up and running.

Installing the helm chart

Running Local

Install the dependencies using PipEnv.

pipenv install 

activate the Pipenv shell.

pipenv shell

Run the application.

kopf run main.py

Build the Docker image.

docker build -t enforcement . 

Configuration

Enforcement uses the environment variables described in the table below to run locally or in the production and you have the option of create a config.ini to configure as well instead variables and last option is use secret to configure you sources(rancher, gke, eks). You can see the examples below.

Creating an environment variables

Environment Variable	Example	Description
ARGO_URL	https://myargourl.domain.com	Argo URL
ARGO_USERNAME	admin	Argo Username
ARGO_PASSWORD	password	Argo Password
OPERATOR_NAMESPACE	argocd	Operator Namespace

Supported cluster sources

Enforcement aims to detect the creation of clusters in several services of managed Kubernetes and cluster orchestration. Currently, the only cluster source supported is Rancher. We are developing support for EKS, GKE and AKS.

Creating a ClusterRule

See a complete example of creating ClusterRule for clusters created through Rancher.

The enforcements field defines all packages that will be installed in the clusters that match the criteria established within the source.rancher field.

apiVersion: enforcement.globo.com/v1beta1
kind: ClusterRule
metadata:
  name: dev-rules
spec:
  enforcements:
    - name: helm-guestbook
      repo: https://github.com/argoproj/argocd-example-apps
      path: helm-guestbook
      namespace: default
      helm:
        parameters:
          replicaCount: 1
    - name: guestbook
      repo: https://github.com/argoproj/argocd-example-apps
      path: guestbook
  source:
    rancher:
      filters:
        driver: googleKubernetesEngine
      labels:
        cattle.io/creator: "norman"
      ignore:
        - cluster1
        - cluster2
        - cluster3

The rancher.filters, rancher.labels and rancher.ignore fields are specific to Rancher. Other cluster sources may have other values. You can get all the examples of ClusterRules objects here.

A ClusterRule also supports dynamic configurations using the Jinja expression language. You can create dynamic models using cluster fields returned by cluster source, or any other valid Python code.

apiVersion: enforcement.globo.com/v1beta1
kind: ClusterRule
metadata:
  name: dynamic-rule
spec:
  enforcements: 
      #The variable name references the cluster name defined in the Cluster source.
    - name: ${% if name=='cluster1' %} guestbook-cluster1 ${% else %} guestbook-other ${% endif %} 
      repo: https://github.com/argoproj/argocd-example-apps #Git repository
      path: helm-guestbook 
      namespace: ${{ name }} #Cluster name
      helm:
        parameters: 
          replicaCount: ${{ 2*5 }}
          clusterURL: ${{ url }} #Cluster URL
  source:
    rancher: {}

The name, url and id fields are available for all cluster sources, however there are also specific fields for each Cluster source, see the list here.

You can also configure your ClusterRule using HTTP request returns using the Python requests library.

apiVersion: enforcement.globo.com/v1beta1
kind: ClusterRule
metadata:
  name: dynamic-rule
spec:
  enforcements: 
    - name: guestbook
      repo: https://github.com/argoproj/argocd-example-apps 
      path: helm-guestbook 
      helm:
        parameters: 
          uuid: ${{ requests.get('https://httpbin.org/uuid').json()['uuid'] }}
  source:
    rancher: {}

Triggers

You can configure triggers to be notified every time enforcement is installed on a cluster. Triggers are HTTP requests that Enforcement will make every time a package is installed on some cluster selected by cluster rule.

apiVersion: enforcement.globo.com/v1beta1
kind: ClusterRule
metadata:
  name: dev-rule #Rule name
spec:
  enforcements:
   - name: guestbook #Name
     repo: https://github.com/argoproj/argocd-example-apps #Git repository
     path: guestbook #Package folder within the repository
  source:
    rancher: {}
  triggers:
    beforeInstall:
      endpoint: http://myendpoint.com/before
      timeout: 5 #Optional. 5 seconds is the default
    afterInstall:
      endpoint: http://myendpoint.com/after
      timeout: 15 #Optional. 5 seconds is the default

Creating a Secret

Enforcement obtains the credentials to connect to the cluster source through a secret that must be previously created. See an example of a secret created for the Rancher Cluster Source.

apiVersion: v1
kind: Secret
metadata:
  name: rancher-secret
type: Opaque
stringData:
  token: abc94943#434!
  url: https://rancher.test.com

In this example of a secret for Rancher, we need to put two values that are the url and token however for other sources it could be different. You can see others example of secret objects here.

You can specify the name of the secret in the spec.source.secretName field of the cluster source object. When no secret name is specified, Enforcement looks for a secret with the same name as the source cluster used.

HOW TO TEST

To run the tests without coverage you may call:

make test

To run the tests with coverage you may call and before called make test:

make coverage 

To generate the html you may call and called make test and make coverage before:

make generate

enforcement's Issues

Optimizations

It was found that Enforcement makes many requests to ArgoCD to verify the packages installed in each cluster. Ideally a request should be made, to bring in all installed packages and filter packages within the application.

It is also possible to optimize the registration of clusters, registering only those that are not yet registered in ArgoCD.

Create a project on ArgoCD for each cluster.

When Enforcement detects a new cluster and registers on ArgoCD, it must also create a new project on ArgoCD with the same name as the created cluster.
In this way, specific packages for a given cluster can be installed within the project.
When Enforcement detects that a cluster has been removed, its project must also be deleted.

Error checking cluster status

At some points in the Enforcement, it is necessary to check the status of the cluster. There is currently a problem verifying this status. The Enforcement is checking the "Cluster" content, when it should check if the "Cluster" field exists.

GitHub Actions workflow failing

There is a problem with the GitHub Actions workflow. This problem occurs when installing dependencies via pipenv and causes the workflow to fail.

Dynamic configuration

Enforcement must allow dynamic values ​​in the configuration of cluster rules.
It should be possible to specify in the configuration of the cluster rule, cluster information returned by the cluster source through a simple expression language.

Example:

apiVersion: enforcement.globo.com/v1beta1
kind: ClusterRule
metadata:
  name: dev-rules
spec:
  enforcements:
    - name: helm-guestbook
      repo: https://github.com/argoproj/argocd-example-apps
      path: "${cluster_name}/helm-guestbook"
      helm:
        parameters:
          replicaCount: 1
          cluster: ${cluster_name}

  source:
    rancher: {}

Dynamic configuration should only be allowed within the enforcement field

There is no Requirements.txt file or Setup.py

if you wish I may add them.
Greetings to the wonderful globocom team from my side.

GKE Cluster Source

The purpose of this task is to implement support for the GKE (Google Kubernetes Engine) cluster source.

The access information as a service account token will be passed on to the secret that the cluster source must know how to read.

The custom resource that contains the GKE source cluster should look like this:

  source:
    secretName: gke-secret
    gke:
      filters: {}
      tags: {}
      ignore:
        - cluster1
        - cluster2
        - cluster3 

The cluster source must allow in the filters field, all filters allowed in the command gcloud container clusters list.
The rules for reading the secret performed by the cluster source must follow the requirements specified in issue #4.

Trigger BeforeInstall and AfterInstall

The application must allow the configuration of triggers that must be activated before and after each package installation performed. The triggers must be HTTP requests of the Post type, where information about the cluster that is receiving the installation and the respective package must be passed.

Below is an example of a cluster rule with a configured trigger.

apiVersion: enforcement.globo.com/v1beta1
kind: ClusterRule
metadata:
  name: dev-rules
spec:
  enforcements:
    ...
  source:
    secretName: rancher-secret
    rancher: {}
  triggers:
    beforeInstall: 
      endpoint: http://localhost:8080/before
      timeout: 5 # in seconds 
    afterInstall:
      endpoint: http://localhost:8080/after
      timeout: 15 # in seconds   
      
      ...

In the absence of a timeout setting, the default should be 5 seconds.

Improve error handling

This issue describes a way to improve the error handling of the enforement operator.

The following situations need to be addressed:

• Invalid enforcement
• Connection error with the cluster source
• Connection error with ArgoCD

When an attempt is made to create or update an invalid enforcement, it must be listed in the status of the custom resource.

See an example of the status

Status:
  Create:
    Clusters:
      Name:  cluster1
      URL:   https://192.168.0.76/k8s/clusters/c-cx9wq
    install_errors:
      helm-guestbook
  Kopf:
    Progress:
  Sync:
    Clusters:
      Name:  cluster1
      URL:   https://192.168.0.76/k8s/clusters/c-cx9wq
  update/spec.enforcements:
    install_errors:
    update_errors:

The operator should also not allow the sync controller to start if the create has ended in error.

The current status of the custer rule object is never read

The controller sync method is called periodically by kopf's @Timer in order to check if there are new clusters in the sources clusters. However, I found that after a while, kopf no longer passes the most current version of the status to the sync method. This kopf bug is preventing Enforcement from working properly.

Log warning

Kopf is logging a warning message whenever a controller makes a status change. The message says "Paching failed with inconsistencies".
This warning does not interfere with the operation of the operator, but it would be good to understand the cause of this message and correct this problem.

Configuration secrets for cluster sources

Currently, the access token and the Rancher's URL are reported to Enforcement through variables from the RANCHER_URL and RANCHER_TOKEN environment variables.

As the Enforcement has the objective of being independent from the cluster orchestration service, it would be better if these values were saved as secrets in the cluster and the name of the secret was referenced in the created ClusterRule object.

Example:

apiVersion: enforcement.globo.com/v1beta1
kind: ClusterRule
metadata:
  name: dev-rules
spec:
  enforcements:
    ...
  source:
    secretName: rancher-secret
    rancher: {}
      
      ...

Preferably, if no secretName is specified, Enforcement should look for the secret with the same name as the source.
This standard should be adopted in the next developed sources.

Change the control layer

We identified many problems in the framework used for the development of operators used in the project.

Kopf has been presenting serious problems in the @kopf.time functionality and these problems are affecting the functioning of Enforcement.
This task aims to find and implement another solution for the control layer, replacing it with another framework or implementing a vanilla solution using the Kubernetes sdk.

Change Rancher Source cluster search strategy

Currently the rancher source searches for all clusters with active status and ignores the others. This search strategy causes Enforcement to remove apps when the cluster changes to any status other than active.
The Rancher cluster source must return all clusters that have different status than provisioning and waiting.

Unit tests

Implement unit tests at the project's domain layer.
Refactors required to ensure testability should also be made.

Creation of the CI process in GitHub Actions

Task with the objective of automating the build and upload of the enforcement docker image to the Docker Hub whenever a new versioning tag is created.

EKS Cluster Source

The purpose of this task is to implement support for the origin of the EKS (Elastic Kubernetes Service) cluster.

Access information as a service account token will be passed on to the secret that the cluster source must know how to read.

The custom resource that contains the EKS source cluster should look like this:

  source:
    secretName: eks-secret
    eks:
      tags: {}
      ignore:
        - cluster1
        - cluster2
        - cluster3 

The rules for reading the secret performed by the cluster source must follow the requirements specified in issue #4

Test exception cases

The objective of this task is to increase the coverage of tests in the domain layer, adding the cases of exception listed in the issue #15.

Error deleting resources

Enforcement is throwing an exception when multiple cluster rules try to delete the same resources as clusters, projects and applications. Enforcement should ignore 404 errors returned by the ArgoCD API when removing these resources.