traefik / mesh

Traefik Mesh - Simpler Service Mesh

Traefik Mesh: Simpler Service Mesh

Traefik Mesh is a simple, yet full-featured service mesh. It is container-native and fits as your de-facto service mesh in your Kubernetes cluster. It supports the latest Service Mesh Interface specification SMI that facilitates integration with pre-existing solution. Moreover, Traefik Mesh is opt-in by default, which means that your existing services are unaffected until you decide to add them to the mesh.

Non-Invasive Service Mesh

Traefik Mesh does not use any sidecar container but handles routing through proxy endpoints running on each node. The mesh controller runs in a dedicated pod and handles all the configuration parsing and deployment to the proxy nodes. Traefik Mesh supports multiple configuration options: annotations on user service objects, and SMI objects. Not using sidecars means that Traefik Mesh does not modify your Kubernetes objects, and does not modify your traffic without your knowledge. Using the Traefik Mesh endpoints is all that is required.

Prerequisites

To run this app, you require the following:

• Kubernetes 1.11+
• CoreDNS installed as Cluster DNS Provider (versions 1.3+ supported)
• Helm v3

Install (Helm v3 only)

helm repo add traefik https://traefik.github.io/charts
helm repo update
helm install traefik-mesh traefik/traefik-mesh

You can find the complete documentation at https://doc.traefik.io/traefik-mesh.

Contributing

Contributing guide.

mesh's Issues

Code de-dupe

There seems to be a bunch of duplicated code in the meshcontroller handler.

Look into using interfaces to allow for dedupe.

Review CRD for SMI

SMI is implemented via CRD, we need to get a controller for that

Create Dockerfile and build for i3o

This needs to run in a docker container.

We need a dockerfile to make this work.

Standardize project layout

Based on https://github.com/golang-standards/project-layout,

we have a non-standard layout.

We should improve by moving code into the standardized directories now

Dashboard

Did we want a dashboard for this project? Or did we want to do something like have a deployable prometheus + grafana container sort of thing (like we do with our advocacy demos?)

Route by service name and Service IP

We should route by not only service name (DNS) but also service IP.

This is for both TCP and HTTP

Service load balancing

We need to have load balancing in Traefik v2 before we can leverage it in i3o

Add Patch as a job with a separate serviceaccount in helm

We need:

1. A job
2. A new service account
3. New permissions via RBAC for this job

Use TCP routing by default

Since services are TCP by default, we should also use TCP routing by default

Controller improvement/consolidation

Currently we use separate controllers for watching each type.

Once we get further along, we may not need separate controllers for each type, or we may be able to consolidate these controllers in a more efficient manner.

Tracing Implementation

How did we want to implement tracing?

Not sure how tracing is implemented in Traefik v2

Features and Integration tests

The following features and integration tests are needed:

• TCP Routing
  • Feature
  • Integration test
• HTTP Routing
  • Feature
  • Integration test
• Metrics - #65
  • Feature
  • Integration test
• Tracing - #63
  • Feature
  • Integration test
• Service load balancing - #64
  • Feature
  • Integration test
• Routing Rules - This one could be combined with TCP and HTTP, since they will both use rules, however this is also possibly done part of the traffic routing part of SMI
  • Feature
  • Integration test
• Retries - #166
  • Feature
  • Integration test
• Failover
  • Feature
  • Integration test
• Access control
  • Feature
  • Integration test
• Rate limit - #168
  • Feature
  • Integration test
• circuit breakers - #167
  • Feature
  • Integration test

Create integration test to handle a request

We should have an integration test that can handle an end-to-end request

Add required Match and Kind in ingressRoute creation

Match and Kind are not optional. They need to be defined.

Create helm chart for i3o

We want to use helm to deploy i3o

We need a base helm chart to configure the install

Separate clusterRoleBinding into rolebinding

Many of the operations in the controller RBAC are due to updates in the mesh namespace

It might be nice to separate out the mesh namespace RBAC into a roleBinding,

and keep a much more minimal ClusterRoleBinding, since we don't need global permissions for everything in the cluster

Define label structure for configuration on services

We should define service labels to assist with defining routing.

This would allow us to start work on implementing features

IngressRouteTCP has no middlewares

This means that IP filtering and whitelisting are not options that we have available to us.

I am going to speak with @juliens and @ldez about extending this with our provider in our i3o traefik implementation.

Investigate Reverse DNS lookup on IP to get pod for SMI

We could look into using reverse IP lookups on the source IP to get the pod reference.

this may allow us to not have to deal with IP whitelisting for source filtering

Convert from file configuration -> KubernetesCRD for config source

Instead of using a templated config file, we should move to CRD, which don't require templating files

Add ability to create KubernetesCRD from Traefik CE

We need to be able to CRUD the Traefik CE KubernetesCRDs

Demo Data

Do we need to have demo data in the app anymore?

Metrics implementation

Once metrics are enabled in traefik v2, we need to have them implemented in i3o. This will also be required for SMI

Move traefik mesh pods from http -> tcp mode

We don't want to use http for these pods.

We need to use TCP

SMI Traffic-Split Implementation

Traffic-split should be fairly easy to accomplish, once we have some weighting etc from Traefik v2.

Until then, we could do a janky workaround where we add the same route x/y times to mathematically make it work, but that is terrible

Create label for ingressRoute selection to avoid external routing objects

We should use:

i3o-mesh:internal as the label

Implement Prometheus, Grafana and Jaeger into i3o helm chart

We need to have the helm chart install:

• Prometheus
• Grafana
• Jaeger

And allow them to be configured from a default configuration to external sources if desired

How to define static config for I3o mesh nodes

Some of the traefik features (such as metrics) are static.

Did we want to manage these by a configmap (toml file)? or did we want to use CLI for everything?

CLI is so much cleaner than having to hunt down configmaps. We also don't have to worry about changes not propagating if we use CLI.

Rename Verify methods to CRUD equivalents

Although verifiyExists is nice,

create is clearer. With a comment that if exists, returns the object.

Related to #57

Documentation

Eventually we are going to need documentation,

Because people.

Create Readme for installation

We need to have a Readme so that it is obvious how to install and configure i3o in a fresh cluster

Improve controller queues

Currently each controller has its own queue for processing events. We may want to have a global controller queue to be able to process so that we can move processing to a higher level controller (mesh controller)

Verify Patch service account is deleted when controller starts

We use a high-perms RBAC SA for patching DNS.

We should confirm that it is deleted as part of the controller verification on startup

Misnamed Imports

Since we have been using sources from different codebases, we need to standardize our imports so that we don't duplicate names.

	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	appsv1 "k8s.io/api/apps/v1"

Handle services with multiple ports

We need to be able to create ingressRoutes(tcp) with different ports for each service port

Core pods all deleted at once

A regression was introduced in #34 that deletes all the core pods in the cluster.

This leads to total DNS outage for the entire cluster (including API).

I think it would be better to add a meta label to force a restart natively instead of deleting the pods.

Update meshservice/IngressRoute[TCP]s on user service update

We have create + delete, but we should handle updating.

Should also be able to have an integration test for this.

Core Functionality

There are a few things that this controller needs:

• Dockerization - #4
• CI integration - #11
• Tests - #29
• Helm chart - #16

Functionality requirements missing:

• Has to be able to CRUD Traefik CE KubernetesCRDs - #30
• Has to be able to interact with SMI CRDs - #66

Nice to haves:

• Be able to verify its own RBAC permissions to check if it has sufficient permissions - Not needed, since we do not want to grant access to rbac. Delegating to helm chart to ensure correct permissions
• Be able to run multiple replicas, and handle these CRUD operations atomically.

Ignore `default/kubernetes` service

The kubernetes API service should not be managed by mesh.

We should add it to the list of namespaces to ignore.

Define MVP features for v1 launch

We should define a set of features that we want for v1 launch, so that we can ensure we have the features ready, and tests written to ensure correct behavior for these features.

Using "=" in flags breaks kubeconfig

Using:

./i3o patch --kubeconfig ~/.kube/config --master 127.0.0.1:9000 --debug
INFO[0000] Building Kubernetes Client...                
INFO[0000] Building Kubernetes CRD Client...            
INFO[0000] Preparing Cluster...         

Works, but:

./i3o patch --kubeconfig=~/.kube/config --master=127.0.0.1:9000 --debug
FATA[0000] Error building clients: stat ~/.kube/config: no such file or directory 

Breaks

Create Service Controller for Access Control

We need a service controller to manage Shadow services for access control.

This will allow SMI implementation of Access Control, and will may allow more features to be implemented in TrafficSplit etc.

Remove Traefik-mesh deployment code from controller

Since we are going to use helm, remove this code from controller

Implement the traefik CRD registration in helm chart

Update CoreDNS patching mechanism

Right now we do a string replacement to patch in our coreDNS config.

We should look at having our own server block that we can just append to the data.

This would allow us to be compatible on more systems without risk of breaking things.

And be less janky.

Discuss testing for i3o

How do we want to do tests?

Move Patch-DNS to a new command

We need to move the Patch DNS code to a new command, so that we can use separate RBAC permissions + jobify it.

SMI Traffic-Access-Control Implementation

So it appears that traffic control via SMI is done by checking Service Accounts of running pods and tying them to destinations with running service accounts.

This is not difficult to do, but there are a bunch of ways we can accomplish it with traefik.

We can either do an IP whitelist, or we can do some other sort of filtering.

I have a few concerns:

1. If we are running in a large network, we may run into issues with large whitelists/blacklists
2. If we have a large dynamic network, there may be lots of changes that we have to watch for, and that may be a lot of load.

These concerns may not be big in reality, but just something to think about

Follow up RBAC Updates for patch/controller

We need to verify that the RBAC matches the client access requirements

Log warning when UDP port is detected

Since we don't have UDP support at this time, we should be logging a warning, and skipping the port