In this lab we'll demonstrate basic networking concepts in Kubernetes using the guestbook application.
In this lab you'll learn:
- How pods communicate with each other
- How pods are exposed to the internet
- How traffic between pods can be restricted
This lab assumes that the reader is familiar with basic Kubernetes concepts. See the IBM Cloud Kubernetes Service Lab for a refresher of these concepts.
Before you begin, you need to install the required CLIs to manage your Kubernetes clusters. IBM provides an installer here to get all of these tools together. There are instructions for how to obtain the tools manually if desired. The following tools are used in this lab:
- kubectl CLI
kubectl
is a command line interface for running commands against Kubernetes clusters.
- ibmcloud CLI
ibmcloud
is a command line interface for managing resources in IBM Cloud.
This lab depends upon features such as an ingress controller which require a standard Kubernetes cluster. In order to create a standard cluster you must have either a Pay-As-You-Go or a Subscription IBM Cloud account. See https://cloud.ibm.com/docs/account/index.html#accounts for further information about account types.
The application used in this lab is a simple guestbook website where users can post messages. You should clone it to your workstation since you will be using some of the configuration files.
$ git clone https://github.com/IBM/guestbook.git
It is recommended to use the IBM Cloud dashboard to create a standard cluster because it will help guide you through the configuration options and it will show you the estimated cost.
- Click the login button in the upper right and follow the login prompts to log into your account.
- Click the
Containers
tab on the left side of the window and then select the "Kubernetes Service" tile. - Click the
Create
button. - Fill in the form that appears. First select a location for your cluster as this will dictate the other options. Then choose a zone within the location and a machine type. Set the number of worker nodes to 2. Give a name to your cluster; for this lab we'll use "myStandardCluster".
- Review the cost estimate on the right side of the window.
- Click the
Create Cluster
button.
Now we'll continue using the command line. Log in to the IBM Cloud CLI and enter your IBM Cloud credentials when prompted.
ibmcloud login
Note: If you have a federated ID, use ibmcloud login --sso
to log in to the IBM Cloud CLI.
Check the status of your cluster as follows.
$ ibmcloud ks clusters
OK
Name ID State Created Workers Location Version
myStandardCluster fc5514ef25ac44da9924ff2309020bb3 normal 12 minutes ago 2 Dallas 1.10.7_1520
If the cluster state is pending
, wait for a moment and try the command again.
Once the cluster is provisioned (state is normal
), the kubernetes client CLI kubectl
needs to be configured to talk to the provisioned cluster.
Run ibmcloud ks cluster-config myStandardCluster
which will create a config file on your workstation.
$ ibmcloud ks cluster-config myStandardCluster
OK
The configuration for mycluster was downloaded successfully. Export environment
variables to start using Kubernetes.
export KUBECONFIG=/home/gregd/.bluemix/plugins/container-service/clusters/myStandardCluster/kube-config-hou02-myStandardCluster.yml
Copy the export
statement that you get and run it. This sets the KUBECONFIG
environment variable to point to the kubectl config file.
This will make your kubectl
client work with your new Kubernetes cluster. You can verify that by entering a kubectl
command.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
10.177.184.185 Ready <none> 2m v1.10.7+IKS
10.177.184.220 Ready <none> 2m v1.10.7+IKS
The following figure depicts the components of the sample guestbook application.
The guestbook application is a Go program which runs inside a container that is deployed to the Kubernetes cluster. It includes a simple HTML page and javascript file for a browser front-end. It uses Redis to store its data. It writes data to a Redis master instance and reads data from multiple Redis slave instances. The master and slave instances also run as containers that are deployed to the Kubernetes cluster.
Let's first use the configuration files in the cloned Git repository to deploy the containers and create services for them. We'll then explore the resulting environment.
$ cd guestbook/v1
$ kubectl create -f redis-master-deployment.yaml
deployment.apps/redis-master created
$ kubectl create -f redis-master-service.yaml
service/redis-master created
$ kubectl create -f redis-slave-deployment.yaml
deployment.apps/redis-slave created
$ kubectl create -f redis-slave-service.yaml
service/redis-slave created
$ kubectl create -f guestbook-deployment.yaml
deployment.apps/guestbook-v1 created
$ kubectl create -f guestbook-service.yaml
service/guestbook created
Let's look at the pods that were created.
The guestbook-deployment.yaml
file requested 3 replicas.
The redis-master-deployment.yaml
file requested a single replica.
The redis-slave-deployment.yaml
file requested 2 replicas.
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
guestbook-v1-7fc76dc46-bl7xf 1/1 Running 0 2m 172.30.108.141 10.177.184.220
guestbook-v1-7fc76dc46-ccfgs 1/1 Running 0 2m 172.30.58.207 10.177.184.185
guestbook-v1-7fc76dc46-g7hq2 1/1 Running 0 2m 172.30.108.142 10.177.184.220
redis-master-5d8b66464f-p76z8 1/1 Running 0 4m 172.30.108.139 10.177.184.220
redis-slave-586b4c847c-ll9gc 1/1 Running 0 4m 172.30.108.140 10.177.184.220
redis-slave-586b4c847c-twjdb 1/1 Running 0 4m 172.30.58.206 10.177.184.185
There are two networks:
-
The nodes (10.177.184.220, 10.177.184.185) exist on a private VLAN which is part of your infrastructure account. (They also exist on a public VLAN which we'll discuss later.)
-
The pods (172.30.108.141, 172.30.108.142, etc.) exist on a virtual network which is created by Kubernetes (technically by a container networking plugin to Kubernetes called Calico). Each pod is assigned a unique IP and can talk to any other pod using its IP.
You can observe how the pod network operates over the host node network by running a traceroute command in one of the pods. (Note: The ability to run Linux commands within a container varies depending on the base image it uses. This command will work with guestbook but may not work with other containers.)
$ kubectl exec -it guestbook-v1-7fc76dc46-bl7xf -- traceroute 172.30.58.206
traceroute to 172.30.58.206 (172.30.58.206), 30 hops max, 46 byte packets
1 10.177.184.220 (10.177.184.220) 0.007 ms 0.005 ms 0.004 ms
2 10.177.184.185 (10.177.184.185) 0.652 ms 0.360 ms 0.510 ms
3 172.30.58.206 (172.30.58.206) 0.591 ms 0.347 ms 0.499 ms
Here we're running a traceroute command on pod guestbook-v1-7fc76dc46-bl7xf
which is running on node 10.177.184.220
.
We ask it to trace the route to another pod, redis-slave-586b4c847c-twjdb
which has a pod IP address of 172.30.58.206
.
We can see the path goes from the first's pod node over to the second pod's node and into the second pod.
This is accomplished through the use of virtual ethernet interfaces and updates to Linux routing tables.
One benefit of assigning each pod its own IP address is that applications can use standard ports (80, 443, etc) without the need to remap them at the node level to avoid port conflicts.
Although pods can communicate with each other using their IP addresses, we wouldn't want application programs to use them directly. A pod's IP address can change each time the pod is recreated. If the pod is scaled, then the set of IP addresses to communicate with can be changing frequently.
This is where the ClusterIP services comes in. Let's look at the services we created for the redis master and slaves.
$ kubectl describe service redis-master
Name: redis-master
Namespace: default
Labels: app=redis
role=master
Annotations: <none>
Selector: app=redis,role=master
Type: ClusterIP
IP: 172.21.193.142
Port: <unset> 6379/TCP
TargetPort: redis-server/TCP
Endpoints: 172.30.108.139:6379
Session Affinity: None
Events: <none>
$ kubectl describe service redis-slave
Name: redis-slave
Namespace: default
Labels: app=redis
role=slave
Annotations: <none>
Selector: app=redis,role=slave
Type: ClusterIP
IP: 172.21.60.238
Port: <unset> 6379/TCP
TargetPort: redis-server/TCP
Endpoints: 172.30.108.140:6379,172.30.58.206:6379
Session Affinity: None
Events: <none>
A ClusterIP service provides a stable virtual IP address which distributes TCP connections (or UDP packets) to a targeted set of pods (called endpoints).
Here we can see that the redis-master service has the virtual IP address 172.21.193.14
and that it distributes requests to the redis-master's pod IP address 172.30.108.139
.
The redis-slave service has a virtual IP address 172.21.60.238
and it distributes requests to the redis-slave's pod IP addresses 172.30.108.140
and 172.30.58.206
.
The method by which Kubernetes implements the virtual IP address varies by Kubernetes release. In the 1.10 release used in this lab the default method is to use iptables to translate (Network Address Translation) the virtual IP addresses to the pod IP addresses and the choice of pod IP is random.
Kubernetes provides a DNS entry for each service so services can be addressed by name instead of IP address.
Let's observe this by doing an nslookup
command from within the container.
(Note: The ability to run Linux commands within a container varies depending on the base image it uses.
This command will work with guestbook but may not work with other containers.)
$ kubectl exec -it guestbook-v1-7fc76dc46-bl7xf -- nslookup redis-master
Server: 172.21.0.10
Address 1: 172.21.0.10 kube-dns.kube-system.svc.cluster.local
Name: redis-master
Address 1: 172.21.193.142 redis-master.default.svc.cluster.local
Here we see that the name redis-master
is resolved to address 172.21.193.142
which is the virtual IP address of the redis-master
service.
Services are assigned a DNS name of the form <service>.<namespace>.svc.cluster.local
.
The namespace is needed to address services across namespaces.
In this lab we are only using the default
namespace so using the service name alone is fine to find services.
The domain name svc.cluster.local
does not need to be specified inside the pod because Kubernetes sets this
in the domain search path in the pod's /etc/resolve.conf
file.
C:\>kubectl exec -it guestbook-v1-7fc76dc46-bl7xf -- cat /etc/resolv.conf
nameserver 172.21.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
The types of IPs presented so far, pod IPs and ClusterIPs, are usable only from within the Kubernetes cluster. It is not possible for applications outside the cluster to use them to reach a pod (without additional configuration, e.g. adding your own routes). For that we need to use a type of service which provides an external IP address. Kubernetes provides two service types which do this.
-
A
NodePort
service exposes the service at a static port (the NodePort) on each node. A ClusterIP service, to which the NodePort service will route, is automatically created. -
A
LoadBalancer
service exposes the service externally using a cloud provider's load balancer. NodePort and ClusterIP services, to which the external load balancer will route, are automatically created.
Let's take a look at the service which we created for the guestbook application.
$ kubectl describe service guestbook
Name: guestbook
Namespace: default
Labels: app=guestbook
Annotations: <none>
Selector: app=guestbook
Type: LoadBalancer
IP: 172.21.189.71
LoadBalancer Ingress: 169.46.35.163
Port: <unset> 3000/TCP
TargetPort: http-server/TCP
NodePort: <unset> 30347/TCP
Endpoints: 172.30.108.141:3000,172.30.108.142:3000,172.30.58.207:3000
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 20s service-controller Ensuring load balancer
Normal EnsuredLoadBalancer 20s service-controller Ensured load balancer
This is a LoadBalancer type of service. The services build upon each other so a LoadBalancer service is also a NodePort service and a ClusterIP service. Let's break this down.
- A ClusterIP (identified by the
IP
label) was assigned and it is172.21.189.71
. - A NodePort was assigned and it is 30347.
- A LoadBalancer IP address was assigned and it is
169.46.35.163
.
We now have two ways to access the guestbook application from outside the cluster, either through a node's IP address or through the load balancer's IP address.
The latter is straightforward: use the loadbalancer address and the port that's configured by the service, which in this case is 3000
.
http://169.46.35.163:3000
Using the NodePort service requires us to first find the public addresses of the worker nodes.
$ ibmcloud ks workers myStandardCluster
OK
ID Public IP Private IP Machine Type State Status Zone Version
kube-dal10-crfc5514ef25ac44da9924ff2309020bb3-w1 169.47.252.42 10.177.184.220 u2c.2x4.encrypted normal Ready dal10 1.10.7_1520*
kube-dal10-crfc5514ef25ac44da9924ff2309020bb3-w2 169.48.165.242 10.177.184.185 u2c.2x4.encrypted normal Ready dal10 1.10.7_1520*
We need to combine the worker node's IP address with the NodePort assigned by Kubernetes which in this example is 30347
. The NodePort gives
the service a unique endpoint on the node.
http://169.47.252.42:30347
http://169.48.165.242:30347
Note that a NodePort service also distributes TCP connections to the pods. It does not require a pod to be running on the node which you address. If there is a pod running on the node which you address, it is not always the case that a connection will be routed to that pod if there are pods on other nodes as well. The connection might be routed to one of those other pods.
The NodePort service typically would be used only in the following cases:
- in cloud environments which don't support a LoadBalancer service (such as in a trial account on IBM Cloud or in a local environmment such as
minikube
); - in development environments where you don't need a LoadBalancer service.
A Kubernetes LoadBalancer service is a TCP layer (layer 4) load balancer. If you want the features of an application layer (layer 7) load balancer, you need to use an Ingress resource instead of a LoadBalancer service. Let's change the guestbook application for using a LoadBalancer service to using an Ingress resource. First let's delete the existing guestbook service.
$ kubectl delete service guestbook
service "guestbook" deleted
We'll use the following yaml to define a new service and Ingress resource.
apiVersion: v1
kind: Service
metadata:
name: guestbook
labels:
app: guestbook
spec:
ports:
- port: 3000
targetPort: http-server
selector:
app: guestbook
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: guestbook
annotations:
ingress.bluemix.net/rewrite-path: "serviceName=guestbook rewrite=/"
spec:
tls:
- hosts:
- <ingress-subdomain>
secretName: <ingress-secret>
rules:
- host: <ingress-subdomain>
http:
paths:
- path: /guestbook/
backend:
serviceName: guestbook
servicePort: 3000
You can find this yaml here. Download it and open it in a text editor.
There are a couple of updates that need to be made to the yaml before we can give it to Kubernetes. Run the following command:
$ ibmcloud ks cluster-get myStandardCluster
Retrieving cluster myStandardCluster...
OK
Name: myStandardCluster
ID: fc5514ef25ac44da9924ff2309020bb3
State: normal
Created: 2018-09-18T13:58:23+0000
Location: dal10
Master URL: https://169.60.128.2:24627
Master Location: Dallas
Ingress Subdomain: mystandardcluster.us-south.containers.appdomain.cloud
Ingress Secret: mystandardcluster
Workers: 2
Worker Zones: dal10
Version: 1.10.7_1520
Owner Email:
Monitoring Dashboard: -
There are two values of interest in the output (the values in your output may be different):
-
Ingress Subdomain: IBM provides this domain name which you can use to access your application from the internet.
-
Ingress Secret: IBM provides certificates for the IBM-provided domain name. The certificates and private key are stored in this Kubernetes secret.
There are ways to use your own domain name and certificates but for this lab we'll use the ones provided by IBM.
Update your copy of the yaml file to use the values you obtained.
Replace <ingress-subdomain>
with the Ingress Subdomain and replace <ingress-secret>
with the Ingress Secret.
Save the file and then tell Kubernetes to create the resources defined by your copy of the yaml file.
$ kubectl create -f <location-of-your-copy-of-yaml-file>
service/guestbook created
ingress.extensions/guestbook created
Let's walk through what happens with the Ingress resource. Behind the scenes your cluster contains an application
load balancer which in the IBM Cloud is provided by nginx
. A Kubernetes component called an Ingress controller takes
your Ingress resource and translates it to a corresponding application load balancer configuration.
The Ingress resource needs to tell the load balancer which services to expose and how to identify which request should be routed to which service. If you have only one service that will be ever be exposed, this is relatively easy: all requests to your ingress subdomain should be routed to that one service. However it's likely that you'll have more than one service. Furthermore most web applications are written to serve files starting at the root path (/) so a way is needed to separate them. The Ingress resource we're using accomplishes this as follows:
-
It defines a
path
which tells the load balancer that a request that begins with/guestbook/
should be routed to the guestbook service. -
It uses an annotation
ingress.bluemix.net/rewrite-path
which tells the load balancer that when sending requests to the guestbook application, it needs to rewrite the path to replace/guestbook/
with/
.
This pattern can be repeated as you expose more services to the internet.
This diagram shows how the ingress works in this example. There are two nginx pods for high availability.
Kubernetes network policies specify how pods can communicate with other pods and with external endpoints. Default network policies are set up by the IBM Kubernetes service to secure your Kubernetes cluster. If you have unique security requirements, you can create your own network policies.
The following network traffic is allowed by default:
- A pod accepts external traffic from any IP address to its NodePort or LoadBalancer service or its Ingress resource.
- A pod accepts internal traffic from any other pod in the same cluster.
- A pod is allowed outbound traffic to any IP address.
Network policies let you create additional restrictions on what traffic is allowed. For example you may want to restrict external inbound or outbound traffic to certain IP addresses.
For this lab we'll use a network policy to restrict traffic between pods. Let's say that we want to limit access to the redis servers to just the guestbook application. First we can observe that the redis servers are open to any pod by spinning up a Linux shell inside a pod and making a network connection to the redis servers' IP addresses and ports.
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
guestbook-v1-7fc76dc46-bl7xf 1/1 Running 0 4d 172.30.108.141 10.177.184.220
guestbook-v1-7fc76dc46-ccfgs 1/1 Running 0 4d 172.30.58.207 10.177.184.185
guestbook-v1-7fc76dc46-g7hq2 1/1 Running 0 4d 172.30.108.142 10.177.184.220
redis-master-5d8b66464f-p76z8 1/1 Running 0 4d 172.30.108.139 10.177.184.220
redis-slave-586b4c847c-ll9gc 1/1 Running 0 4d 172.30.108.140 10.177.184.220
redis-slave-586b4c847c-twjdb 1/1 Running 0 4d 172.30.58.206 10.177.184.185
$ kubectl run -i --tty --rm busybox --image=busybox -- sh
If you don't see a command prompt, try pressing enter.
/ # nc -v -z 172.30.108.139 6379
172.30.108.139 (172.30.108.139:6379) open
/ # nc -v -z 172.30.108.140 6379
172.30.108.140 (172.30.108.140:6379) open
/ # nc -v -z 172.30.58.206 6379
172.30.58.206 (172.30.58.206:6379) open
/ # exit
Session ended, resume using 'kubectl attach busybox-5858cc4697-hb6zs -c busybox -i -t' command when the pod is running
deployment.apps "busybox" deleted
First we used the kubectl get pods -o wide
command to see the IP addresses assigned to the redis pods.
Then we ran a busybox image to try nc
commands to each of those addresses.
(The redis server ports are defined in the redis-master-service.yaml
and redis-slave-service.yaml
files we used earlier.
The nc
commands show that connections are allowed.
Now let's define a network policy to restrict access to the redis servers to the guestbook application.
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: redis-policy
spec:
podSelector:
matchLabels:
app: redis
ingress:
- from:
- podSelector:
matchLabels:
app: guestbook
You can find this yaml here. Download it to your workstation and tell Kubernetes to create it.
$ kubectl create -f <location-of-your-copy-of-yaml-file>
networkpolicy.networking.k8s.io/redis-policy created
Now we can repeat the nc
commands and see that connections to the redis servers are not allowed from an arbitrary pod.
The guestbook application continues to work though.
$ kubectl run -i --tty --rm busybox --image=busybox -- sh
If you don't see a command prompt, try pressing enter.
/ # nc -v -z 172.30.108.139 6379
nc: 172.30.108.139 (172.30.108.139:6379): Connection timed out
/ # nc -v -z 172.30.108.140 6379
nc: 172.30.108.140 (172.30.108.140:6379): Connection timed out
/ # nc -v -z 172.30.58.206 6379
nc: 172.30.58.206 (172.30.58.206:6379): Connection timed out
/ # exit
Session ended, resume using 'kubectl attach busybox-5858cc4697-mvgsf -c busybox -i -t' command when the pod is running
deployment.apps "busybox" deleted
Continue your learning by visiting the Istio workshop. With Istio, you can manage network traffic, load balance across microservices, enforce access policies, verify service identity, and more.