Consul Service Discovery with Kong Ingress Controller on Kubernetes
Table of Contents
Overview
From Kong API Gateway perspective, using Consul as its Service Discovery infrastructure is one of the most well-known and common integration use cases. This exercise shows how to integrate Kong for Kubernetes (K4K8S), the Kong Ingress Controller based on the Kong API Gateway, with Consul Service Discovery running on a Kubernetes EKS Cluster.
Kong for Kubernetes can implement all sort of policies to protect the Ingresses defined to expose Kubernetes services to external Consumers including Rate Limiting, API Keys, OAuth/OIDC grants, etc.
Architecture
The following diagram describes the Kong for Kubernetes Ingress Controller and Consul Service Discovery implementing a Canary Release:
System Requirements
- A Kubernetes Cluster. This exercise was done on an AWS EKS Cluster. Both Consul Connect and Kong Enterprise support any Kubernetes distribution.
- kubectl
- Helm 3.x
- Consul CLI
- HTTPie and Curl.
Installation
Step 1: Consul Installation
- Add HashiCorp repo to your local Helm installation.
helm repo add hashicorp https://helm.releases.hashicorp.com
- Use the following YAML file to install Consul.
global:
datacenter: dc1
image: "consul:1.8.4"
# Expose the Consul UI through this LoadBalancer
ui:
service:
type: LoadBalancer
# Configure a Consul client on Kubernetes nodes. GRPC listener is required for Connect.
client:
enabled: true
grpc: true
# Minimal Consul configuration. Not suitable for production.
server:
replicas: 1
bootstrapExpect: 1
disruptionBudget:
enabled: true
maxUnavailable: 0
# Sync Kubernetes and Consul services
syncCatalog:
enabled: false
- Create a Namespace for Consul
kubectl create namespace hashicorp
- Install Consul Connect
helm install consul-connect -n hashicorp hashicorp/consul -f consul-values.yml
- Check the installation
$ kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE hashicorp consul-consul-server-0 1/1 Running 0 45s hashicorp consul-consul-xndhd 1/1 Running 0 45s kube-system aws-node-rb528 1/1 Running 0 5m26s kube-system coredns-556765db45-qm7hc 1/1 Running 0 11m kube-system coredns-556765db45-wsctq 1/1 Running 0 11m kube-system kube-proxy-v2mkd 1/1 Running 0 5m26s
$ kubectl get service --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.100.0.1 443/TCP 12m hashicorp consul-consul-dns ClusterIP 10.100.55.251 53/TCP,53/UDP 65s hashicorp consul-consul-server ClusterIP None 8500/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP 65s hashicorp consul-consul-ui LoadBalancer 10.100.193.197 ab45069bc496f4c9a9854a6c7b7d2bb1-1116935595.us-west-1.elb.amazonaws.com 80:32688/TCP 65s kube-system kube-dns ClusterIP 10.100.0.10 53/UDP,53/TCP 11m
Check the Consul Connect services redirecting your browser to Consul UI:
Step 2: Configure Consul DNS
kubectl get service consul-consul-dns -n hashicorp -o jsonpath='{.spec.clusterIP}'
10.100.55.251
kubectl edit configmap coredns -n kube-system
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
consul {
errors
cache 30
forward . 10.100.55.251
}
kind: ConfigMap
metadata:
creationTimestamp: "2020-06-19T13:42:16Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:Corefile: {}
manager: kubeadm
operation: Update
time: "2020-10-19T13:42:16Z"
name: coredns
namespace: kube-system
resourceVersion: "178"
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: 698c5d0c-998e-4aa4-9857-67958eeee25a
Step 3: Deploy Sample Microservice and Canary
Canary description
- Deploy Sample Microservice
Use the following declarations to deploy the Sample Microservice, Sample Deployment and Sample Service
After the deployment you should see the new Kubernetes Pods as well as the new Consul Services.
kubectl apply -f deployment_benigno_v1.yaml kubectl apply -f service_benigno.yaml
- Check the installation
$ kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default benigno-v1-5d5b4b94bf-gbqn7 1/1 Running 0 31s hashicorp consul-consul-server-0 1/1 Running 0 6m32s hashicorp consul-consul-xndhd 1/1 Running 0 6m32s kube-system aws-node-rb528 1/1 Running 0 11m kube-system coredns-556765db45-qm7hc 1/1 Running 0 17m kube-system coredns-556765db45-wsctq 1/1 Running 0 17m kube-system kube-proxy-v2mkd 1/1 Running 0 11m
$ kubectl get service --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default benigno-v1 ClusterIP 10.100.225.125 5000/TCP 47s default kubernetes ClusterIP 10.100.0.1 443/TCP 17m hashicorp consul-consul-dns ClusterIP 10.100.55.251 53/TCP,53/UDP 6m50s hashicorp consul-consul-server ClusterIP None 8500/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP 6m50s hashicorp consul-consul-ui LoadBalancer 10.100.193.197 ab45069bc496f4c9a9854a6c7b7d2bb1-1116935595.us-west-1.elb.amazonaws.com 80:32688/TCP 6m50s kube-system kube-dns ClusterIP 10.100.0.10 53/UDP,53/TCP 17m
- Check the Microservice
Open a terminal and expose the Benigno Service:
kubectl port-forward service/benigno-v1 5000:5000
On another terminal send a request to it:
$ http :5000 HTTP/1.0 200 OK Content-Length: 20 Content-Type: text/html; charset=utf-8 Date: Mon, 19 Oct 2020 20:14:30 GMT Server: Werkzeug/1.0.1 Python/3.8.3 Hello World, Benigno
- Deploy the Canary Release
Use the simliar declarations to deploy the Canary Release Microservice, Canary Deployment and Canary Service
kubectl apply -f deployment_benigno_rc.yaml kubectl apply -f service_benigno_rc.yaml
Step 4: Register the Consul Service for both Microservices releases
- Get the Microservices Services ClusterIP addresses
Since, the two Microservices have been deployed, a new Consul Service, abstracting the two Microservices addresses, should be defined. The addresses can be obtained using the Consul APIs again:
$ kubectl get service --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default benigno-v1 ClusterIP 10.100.225.125 5000/TCP 116s default benigno-v2 ClusterIP 10.100.148.236 5000/TCP 12s default kubernetes ClusterIP 10.100.0.1 443/TCP 18m hashicorp consul-consul-dns ClusterIP 10.100.55.251 53/TCP,53/UDP 7m59s hashicorp consul-consul-server ClusterIP None 8500/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP 7m59s hashicorp consul-consul-ui LoadBalancer 10.100.193.197 ab45069bc496f4c9a9854a6c7b7d2bb1-1116935595.us-west-1.elb.amazonaws.com 80:32688/TCP 7m59s kube-system kube-dns ClusterIP 10.100.0.10 53/UDP,53/TCP 18m
- Create the new Consul Service templates
Using these templates, primary and secondary, create declarations for the Consul Service with the Microservices addresses as primary and secondary. Notice that the new Consul Service is named as benigno1. That's the name the consumers of the service should use. Moreover, the declarations are defining load balancing weights for primary and secondary IPs.
ben0.json
{
"ID": "ben0",
"Name": "benigno1",
"Tags": ["primary"],
"Address": "10.100.225.125",
"Port": 5000,
"weights": {
"passing": 80,
"warning": 1
}
}
and
ben1.json
{
"ID": "ben1",
"Name": "benigno1",
"Tags": ["secondary"],
"Address": "10.100.148.236",
"Port": 5000,
"weights": {
"passing": 20,
"warning": 1
}
}
- Register the new Consul Service
Open a terminal and expose the Benigno Service:
kubectl port-forward service/consul-consul-server -n hashicorp 8500:8500
Open another terminal to register the Consul Service using the Consul API:
http put :8500/v1/agent/service/register < ben0.json http put :8500/v1/agent/service/register < ben1.json
Check the Service with:
$ http :8500/v1/agent/health/service/name/benigno1
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 258
Content-Type: application/json
Date: Mon, 19 Oct 2020 20:18:24 GMT
Vary: Accept-Encoding
X-Consul-Reason: passing
[
{
"AggregatedStatus": "passing",
"Checks": [],
"Service": {
"Address": "10.100.225.125",
"EnableTagOverride": false,
"ID": "ben0",
"Meta": {},
"Port": 5000,
"Service": "benigno1",
"TaggedAddresses": {
"lan_ipv4": {
"Address": "10.100.225.125",
"Port": 5000
},
"wan_ipv4": {
"Address": "10.100.225.125",
"Port": 5000
}
},
"Tags": [
"primary"
],
"Weights": {
"Passing": 1,
"Warning": 1
}
}
},
{
"AggregatedStatus": "passing",
"Checks": [],
"Service": {
"Address": "10.100.148.236",
"EnableTagOverride": false,
"ID": "ben1",
"Meta": {},
"Port": 5000,
"Service": "benigno1",
"TaggedAddresses": {
"lan_ipv4": {
"Address": "10.100.148.236",
"Port": 5000
},
"wan_ipv4": {
"Address": "10.100.148.236",
"Port": 5000
}
},
"Tags": [
"secondary"
],
"Weights": {
"Passing": 99,
"Warning": 1
}
}
}
]
- Register the Kubernetes Service
The Kubernetes service defines an internal reference for the Consul Service:
kind: Service
apiVersion: v1
metadata:
namespace: hashicorp
name: benigno1
spec:
ports:
- protocol: TCP
port: 5000
type: ExternalName
externalName: benigno1.service.consul
Use kubectl to apply it:
kubectl apply -f externalservice_benigno.yaml
Step 5: Kong for Kubernetes (K4K8S) Installation
- Install Kong for Kubernetes
Add Kong Repository:
helm repo add kong https://charts.konghq.com
Create a Namespace:
kubectl create namespace kong
Install Kong for Kubernetes with Helm
helm install kong kong/kong -n kong --set ingressController.installCRDs=false
- Check the installation
$ kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default benigno-v1-5d5b4b94bf-gbqn7 1/1 Running 0 8m48s default benigno-v2-cdc4d9fcd-vjp7d 1/1 Running 0 7m4s hashicorp consul-consul-server-0 1/1 Running 0 14m hashicorp consul-consul-xndhd 1/1 Running 0 14m kong kong-kong-5994fdbfdd-mkv9z 2/2 Running 0 41s kube-system aws-node-rb528 1/1 Running 0 19m kube-system coredns-556765db45-qm7hc 1/1 Running 0 25m kube-system coredns-556765db45-wsctq 1/1 Running 0 25m kube-system kube-proxy-v2mkd 1/1 Running 0 19m
$ kubectl get service --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default benigno-v1 ClusterIP 10.100.225.125 5000/TCP 9m5s default benigno-v2 ClusterIP 10.100.148.236 5000/TCP 7m21s default benigno1 ExternalName benigno1.service.consul 5000/TCP 87s default kubernetes ClusterIP 10.100.0.1 443/TCP 26m hashicorp consul-consul-dns ClusterIP 10.100.55.251 53/TCP,53/UDP 15m hashicorp consul-consul-server ClusterIP None 8500/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP 15m hashicorp consul-consul-ui LoadBalancer 10.100.193.197 ab45069bc496f4c9a9854a6c7b7d2bb1-1116935595.us-west-1.elb.amazonaws.com 80:32688/TCP 15m kong kong-kong-proxy LoadBalancer 10.100.5.9 a8d220af2afef4dbab34d2e277cdcae8-944163116.us-west-1.elb.amazonaws.com 80:30798/TCP,443:32303/TCP 60s kube-system kube-dns ClusterIP 10.100.0.10 53/UDP,53/TCP 25m
- Check the Kong Proxy
Hit Kong Proxy through the Load Balancer provisioned by AWS:
$ http a8d220af2afef4dbab34d2e277cdcae8-944163116.us-west-1.elb.amazonaws.com
HTTP/1.1 404 Not Found
Connection: keep-alive
Content-Length: 48
Content-Type: application/json; charset=utf-8
Date: Mon, 19 Oct 2020 20:22:54 GMT
Server: kong/2.1.4
X-Kong-Response-Latency: 0
{
"message": "no Route matched with those values"
}
The return message is coming from Kong for Kubernetes, saying there's no API defined yet.
- Check Consul DNS
Open a terminal inside K4K8S pod to run check the benigno1 naming resolution with a simple command:
$ kubectl exec -ti kong-kong-5994fdbfdd-mkv9z -n kong -- /bin/sh Defaulting container name to ingress-controller. Use 'kubectl describe pod/kong-kong-5994fdbfdd-mkv9z -n kong' to see all of the containers in this pod. / $ nslookup benigno1.service.consul Server: 10.100.0.10 Address: 10.100.0.10:53 Name: benigno1.service.consul Address: 10.100.225.125 Name: benigno1.service.consul Address: 10.100.148.236 / $ exit
Step 6: Define Kong Ingress
- Create an Ingress using this declaration. Notice it refers to the External Service created previously:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: benignoroute
namespace: hashicorp
annotations:
kubernetes.io/ingress.class: kong
konghq.com/strip-path: "true"
spec:
rules:
- http:
paths:
- path: /benignoroute
backend:
serviceName: benigno1
servicePort: 5000
Use kubectl to apply it:
kubectl apply -f benignoroute.yaml
- Testing the Ingress
$ http a8d220af2afef4dbab34d2e277cdcae8-944163116.us-west-1.elb.amazonaws.com/benignoroute HTTP/1.1 200 OK Connection: keep-alive Content-Length: 36 Content-Type: text/html; charset=utf-8 Date: Mon, 19 Oct 2020 20:36:37 GMT Server: Werkzeug/1.0.1 Python/3.8.3 Via: kong/2.1.4 X-Kong-Proxy-Latency: 30 X-Kong-Upstream-Latency: 2 Hello World, Benigno, Canary Release
Run a loop if you want to see the Canary in action
while [ 1 ]; do curl http://a8d220af2afef4dbab34d2e277cdcae8-944163116.us-west-1.elb.amazonaws.com/benignoroute; sleep 1; echo; done
Contributing
Thank you for your interest in contributing! Please submit issues here for revisions.
Support
- HashiCorp Support: https://support.hashicorp.com
- Kong Support: https://support.konghq.com
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent