lholmquist / spring-boot-istio-circuit-breaker-booster

A Spring Boot application demonstrating Istio's Circuit Breaker.

Istio Circuit Breaker mission

Purpose

Showcase Istio’s Circuit Breaker via a (minimally) instrumented Spring Boot application

Prerequisites

1. Openshift 3.9 cluster

2. Istio 0.7.1 with authentication installed on the aforementioned cluster. To install Istio simply follow one of the following docs:

   1. https://istio.io/docs/setup/kubernetes/quick-start.html

   2. https://istio.io/docs/setup/kubernetes/ansible-install.html

3. Enable automatic sidecar injection for Istio (See this for details)

   In order for Istio automatic sidecar injection to work properly, the following Istio configuration needs to be in place:

   1. The policy field is set to disabled in the istio-inject configmap of the istio-system namespace. This can be checked by inspecting the output of

      oc get configmap istio-inject -o jsonpath='{.data.config}' -n istio-system | grep policy

   2. The istio-sidecar-injector MutatingWebhookConfiguration should not limit the injection to properly labeled namespaces. If Istio was installed using the default settings, then make sure the output of

      oc get MutatingWebhookConfiguration istio-sidecar-injector -o jsonpath='{.webhooks[0].namespaceSelector}' -n istio-system

      is empty. It is advised however that you inspect the output of

      oc get MutatingWebhookConfiguration istio-sidecar-injector -o yaml

      to make sure that no other "filters" have been applied.

4. Login to the cluster with the admin user

Environment preparation

Create a new project/namespace on the cluster. This is where your application will be deployed.

oc new-project <whatever valid project name you want>

Build and deploy the application

With Fabric8 Maven Plugin (FMP)

Execute the following command to build the project and deploy it to OpenShift:

mvn clean fabric8:deploy -Popenshift

Configuration for FMP may be found both in pom.xml and src/main/fabric8 files/folders.

With Source to Image build (S2I)

Run the following commands to apply and execute the OpenShift templates that will configure and deploy the applications:

find . | grep openshiftio | grep application | xargs -n 1 oc apply -f

oc new-app --template=spring-boot-istio-circuit-breaker-greeting -p SOURCE_REPOSITORY_URL=https://github.com/snowdrop/spring-boot-istio-circuit-breaker-booster  -p SOURCE_REPOSITORY_REF=master -p SOURCE_REPOSITORY_DIR=greeting-service
oc new-app --template=spring-boot-istio-circuit-breaker-name -p SOURCE_REPOSITORY_URL=https://github.com/snowdrop/spring-boot-istio-circuit-breaker-booster  -p SOURCE_REPOSITORY_REF=master -p SOURCE_REPOSITORY_DIR=name-service

Use Cases

Access the application

Run the following command to determine the appropriate URL to access our demo. Make sure you access the url with the HTTP scheme. HTTPS is NOT enabled by default:

oc get route spring-boot-istio-circuit-breaker-greeting -o jsonpath='http://{.spec.host}{"\n"}'

Open this URL in a web browser.

Without Istio configuration

Try the application to see the default behavior without an Istio configuration.

1. Click on the "Start" button to issue 10 concurrent requests to the name service.

2. Click on the "Stop" button to stop the requests

3. You can change the number of concurrent requests between 1 and 20.

4. All calls go through as expected.

Circuit Breaker With Istio

Initial setup

1. Apply the RouteRule which is required for the DestinationPolicy to take effect:

   oc create -f istio/route_rule.yml -n $(oc project -q)

2. Try the application again.

   At this point, nothing should have changed in the application behavior.

Initial Circuit Breaker configuration

1. Now apply the initial DestinationPolicy that activates Istio’s Circuit Breaker on the name service, configuring it to allow a maximum of 100 concurrent connections.

   oc create -f istio/initial_destination_policy.yml -n $(oc project -q)

2. Try the application again.

   Since we only make up to 20 concurrent connections, the circuit breaker should not trip.

Restrictive Circuit Breaker configuration

1. Now apply a more restrictive DestinationPolicy, after having remove the initial one:

   oc delete -f istio/initial_destination_policy.yml
   oc create -f istio/restrictive_destination_policy.yml -n $(oc project -q)

2. Try the application again

   Since the Circuit Breaker is now configured to only allow one concurrent connection and by default we are sending 10 to the name service, we should now see the Circuit Breaker tripping open. However, experimentally, we observe that this does not happen in a clear-cut fashion: the circuit is not always open. In fact, depending on how fast the server on which the application is running, the circuit might not break open at all. The reason for this is that is the name service doesn’t do much and thus responds quite fast. This, in turn, leads to not having much concurrency at all.

Optional: fault injection

We could try to increase contention on the name service using Istio’s fault injection behavior by applying a 1-second delay to 50% of the calls to the name service.

1. Delete the original RouteRule and apply a new one to do that:

   oc delete -f istio/route_rule.yml
   oc create -f istio/route_rule_with_delay.yml -n $(oc project -q)

2. Try the application again

   You should observe that this doesn’t seem to change how often the circuit breaks open. This is due to the fact that the injected delay actually occurs between the services. So, in essence, this only time shifts the requests, only increasing concurrency marginally (due to the fact that only 50% of the requests are delayed). This still doesn’t let us observe the circuit breaking open properly.

3. For more comfort, re-activate the original RouteRule since we don’t need the artificial one-second delay:

oc delete -f istio/route_rule_with_delay.yml
oc create -f istio/route_rule.yml -n $(oc project -q)

Simulate load on the name service

• We need to increase contention on the name service in order to have enough concurrent connections to trip open the circuit breaker. We can accomplish this by simulating load on the name service by asking it to introduce a random processing time. To accomplish this:

  1. Stop the requests (if that wasn’t already the case)

  2. Checking the "Simulate load" checkbox

  3. Start the requests.

     You should now observe the circuit breaking open.

Undeploy the application

With Fabric8 Maven Plugin (FMP)

mvn fabric8:undeploy

With Source to Image build (S2I)

oc delete all --all
find . | grep openshiftio | grep application | xargs -n 1 oc delete -f

Remove the namespace

This will delete the project from the OpenShift cluster

oc delete project <your project name>