colemickens / azure-kubernetes-demo

Kubernetes 1.4 on Azure Demo

Kubernetes 1.4 on Azure - Demo/Testing

General Status of Kubernetes on Azure

See azure-kubernetes-status for a high-level status of Kubernetes (1.3.x, and 1.4.x) on Azure.

Demo Overview

This demonstrates:

• Native Azure Cloudprovider support (automatic pod networking, automatic L4 load balancers)
• AzureDisk persistent storage plugin (Thanks to Huamin Chen!)

Demo Features

The yaml file that will be deployed demonstrates some core Kubernetes functionality:

• Service object that abstract over Pods meeting a certain selector.
• Service object that declares a need for an external Azure load balancer.
• Deployment objects that contain a Pod template and ensure a certain replica count are available.
• Pod object (inside the Deployment) that shows:
  • Two containers, sharing a volume mount
  • The volume mount being backed by an Azure Disk
  • One busybox container appending a timestamp to a file on the volume mount
  • One busybox container serving the file on the volume mount over HTTP on port 80

Requirements

• kubectl (v1.4.0 or newer) (linux/amd64) (darwin/amd64)

Deploy a Cluster

Follow the instructions in the kubernetes-anywhere project to deploy a cluster: https://github.com/kubernetes/kubernetes-anywhere/tree/master/phase1/azure/README.md.

NOTE: You should ensure that kubernetes_version is set to v1.4.0 or newer.

phase2.kubernetes_version = v1.4.0

Create a VHD Disk

1. Use the azure-tools container to create an ext4-formatted VHD in an Azure storage account:

   (The script will create everything for you, if missing. It will default to westus2.)

   docker pull docker.io/colemickens/azure-tools:latest
   docker run -it docker.io/colemickens/azure-tools:latest
   
   # (inside the container)
   
   export AZURE_SUBSCRIPTION_ID=6f368760-9ad2-4aef-8ff1-fb038d2e75bf
   export AZURE_RESOURCE_GROUP=colemick-vhds2
   export AZURE_STORAGE_ACCOUNT=colemickvhds2
   export AZURE_STORAGE_CONTAINER=colemickvhds2
   export IMAGE_SIZE=10G
   
   ./make-vhd.sh
   # ....
   # ....
   # VHD_URL=https://colemickvhds2.blob.core.windows.net/colemickvhds2/data-disk-082916103645.vhd
   # (end)

   You can leave the azure-tools container now.

Prove the VHD support works correctly

1. Download test-azure-vhd.yaml from this repo:

wget https://raw.githubusercontent.com/colemickens/azure-kubernetes-demo/master/test-azure-disk.yaml

2. Edit in the correct VHD_URL value:

   export VHD_URL=https://colemickvhds2.blob.core.windows.net/colemickvhds2/data-disk-082916103645.vhd
   sed -i "s|VHD_URL|${VHD_URL}|g" ./test-azure-disk.yaml

   (or you can simply do this by hand...)

3. Deploy the test-azure-disk deployment/service:

   kubectl apply -f test-azure-disk.yaml

4. Wait for the pod to start running, and wait for the service to get an external load balancer IP address. It will take some time for the Pod to begin running as the VHD must be attached and mounted first.

   kubectl get pods
   kubectl get services

5. After the pod has become healthy, query the service to see that it has started putting some data on the persistent disk.

kubectl get service test-azure-disk

Connect to the external ip using a browser and check that you see entries.

5. Kill the pod!

kubectl get pods
kubectl delete pod test-azure-disk-XXXXX

6. Kubernetes will schedule a new Pod, potentially on a new VM. This will take some time too as the disk must be unmounted and unattached from the old node, then attached and mounted to the new node.

Wait for it be to healthy again:

kubectl get pods

7. Check the external ip again in your browser and see that the old entries were indeed persisted.

Troubleshooting

1. Make sure the VHD you're using is in a storage account in the same region as the VM.
2. Check kubectl logs --namespace=kube-system kube-controller-manger-<TAB> | grep --text azure to see what important actions took place on the master to create the LB or attach/detach the disk.
3. Check kubectl describe pod test-azure-disk-<TAB> to see the status of the volume mount and the containers in the Pod.

Bonus

1. You can query the cluster nodes with kubectl get nodes and then use kubectl cordon {NODE} before killing the Pod to guarantee that it is not rescheduled onto the same node. (Don't forget to kubectl uncordon {NODE} later!)

Send Feedback

Please feel file issues in the kubernetes-anywhere or kubernetes projects as relevant.

azure-kubernetes-demo's Issues

Issue with mounting vhd

Hi,

I tried having a look at the multiple tickets regardind using mounting vhd volumes, but was unable to understand if the issue is fixed.

I can't see to replicate this demo:

I'm using the following yml replication controller:

apiVersion: v1
kind: Service
metadata:
  name: sayt
  labels:
    run: sayt
spec:
  type: NodePort
  ports:
  - port: 8081
    targetPort: 8081
    protocol: TCP
    name: http
  selector:
    run: sayt

---
apiVersion: v1
kind: ReplicationController
metadata:
  name: sayt
spec:
  replicas: 3
  template:
    metadata:
      labels:
        run: sayt
        uses: elas
    spec:
      containers:
      - name: sayt
        imagePullPolicy: Always
        image: essearch/sayt:dev1
        env:
        - name: SPIFFY_ENV
          value: "azuredev1"
        - name: ES_CLIENT
          value: "10.0.57.114:9200"
        ports:
        - containerPort: 8081
        volumeMounts:
          - name: disk0
            mountPath: "/mnt/disk0"
          - name: disk1
            mountPath: "/mnt/disk1"
          - name: disk2
            mountPath: "/mnt/disk2"
      volumes:
        - name: disk0
          azureDisk:
            diskName: sayt-dev1-node0-osdisk.vhd
            diskURI: https://saytdev1.blob.core.windows.net/strgsayt-dev1/sayt-dev1-node0-osdisk.vhd
        - name: disk1
          azureDisk:
            diskName: sayt-dev1-node1-osdisk.vhd
            diskURI: https://saytdev1.blob.core.windows.net/strgsayt-dev1/sayt-dev1-node1-osdisk.vhd
        - name: disk2
          azureDisk:
            diskName: sayt-dev1-node2-osdisk.vhd
            diskURI: https://saytdev1.blob.core.windows.net/strgsayt-dev1/sayt-dev1-node2-osdisk.vhd
      imagePullSecrets:
      - name: a_key

My pods stop in ContainerCreating status, with the following message:

FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "sayt-j8uwl"/"default". list of unattached/unmounted volumes=[disk0 disk1 disk2]

Here are the logs for kubernetes-controller pod:

E1102 00:48:45.810937       1 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/azure-disk/sayt-dev1-node2-osdisk.vhd\"" failed. No retries permitted until 2016-11-02 00:49:01.810922747 +0000 UTC (durationBeforeRetry 16s). Error: AttachVolume.Attach failed for volume "kubernetes.io/azure-disk/sayt-dev1-node2-osdisk.vhd" (spec.Name: "disk2") from node "sayt-dev1-node-2" with: Attach volume "sayt-dev1-node2-osdisk.vhd" to instance "sayt-dev1-node-2" failed with compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=200 -- Original Error: Long running operation terminated with status 'Failed': Code="AcquireDiskLeaseFailed" Message="Failed to acquire lease while creating disk 'sayt-dev1-node2-osdisk.vhd' using blob with URI https://saytdev1.blob.core.windows.net/strgsayt-dev1/sayt-dev1-node2-osdisk.vhd. Blob is already in use."
I1102 00:49:01.847372       1 reconciler.go:170] Started AttachVolume for volume "kubernetes.io/azure-disk/sayt-dev1-node2-osdisk.vhd" to node "sayt-dev1-node-1"

I'm not sure I understand the error as I'm not attaching the volume anywhere else.

By the way I'm using a recent version of kubernetes-anywhere, and kubectl version shows:

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:45:16Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-alpha.3", GitCommit:"b44b716965db2d54c8c7dfcdbcb1d54792ab8559", GitTreeState:"clean", BuildDate:"2016-08-25T18:31:09Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

phase1.azure.use_ephemeral_drive not found

I'm sorry if this repo shouldn't be used, but I had no luck using other methods for deploying k8s on Azure.

Following instructions as is, fails to deploy the master node because it seems to be missing the value phase1.azure.use_ephemeral_drive.

Adding it to the phase1/azure/Kconfig file (in the dev container) seems to fix the problem, and I was able to deploy a cluster.

Also you have to choose the cluster name to be globally unique, since it gets used to name a storage account.

Add/update demo for AzurePersistentDisk

Augment the existing demo to including a demo of Azure Persistent Disk.

Move Kube-addons documentation to here

Currently, the kubernetes-addons get deployed with polykube in the Polykube demo documentation.

We should move that here, unless kubernetes-anywhere grows addon support before we get to it.

Azure persistent disk not mounted

I've tried your demo, to test both the native loadbalancers and the Azure Disks, and the disks don't seem to mount. FTR I'm using Kubernetes 1.4 and setup the cluster with kubernetes-anywhere with your pr-updates branch.

After creating the VHD with your script and deploying test-azure-disk, I get an IP from the load-balancer but nothing regarding the azure disk:

kubectl --kubeconfig kubeconfig.json describe pod test-azure-disk-349925749-3l7y5
...
Volumes:
  volume-azuredisk:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:

I don't see anything related to the azure disk in the controller manager logs, besides the startup message:

kubectl --kubeconfig kubeconfig.json logs --namespace=kube-system kube-controller-manager-kubeanywhere-master |grep azure-disk
I0928 10:45:51.286094       1 plugins.go:352] Loaded volume plugin "kubernetes.io/azure-disk"
I0928 14:59:00.714628       1 servicecontroller.go:285] Ensuring LB for service default/test-azure-disk
I0928 14:59:00.714804       1 azure_loadbalancer.go:67] ensure(default/test-azure-disk): START clusterName="kubeanywhere" lbName="kubeanywhere"
I0928 14:59:00.963850       1 replica_set.go:482] Too few "default"/"test-azure-disk-349925749" replicas, need 1, creating 1
I0928 14:59:00.968646       1 event.go:217] Event(api.ObjectReference{Kind:"Deployment", Namespace:"default", Name:"test-azure-disk", UID:"1703a90e-858c-11e6-8cc1-000d3a20f494", APIVersion:"extensions", ResourceVersion:"30919", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set test-azure-disk-349925749 to 1
I0928 14:59:00.992948       1 event.go:217] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"test-azure-disk-349925749", UID:"17046208-858c-11e6-8cc1-000d3a20f494", APIVersion:"extensions", ResourceVersion:"30920", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: test-azure-disk-349925749-3l7y5
I0928 14:59:00.997306       1 deployment_controller.go:465] Error syncing deployment default/test-azure-disk: Operation cannot be fulfilled on deployments.extensions "test-azure-disk": the object has been modified; please apply your changes to the latest version and try again
I0928 14:59:01.026695       1 deployment_controller.go:465] Error syncing deployment default/test-azure-disk: Operation cannot be fulfilled on deployments.extensions "test-azure-disk": the object has been modified; please apply your changes to the latest version and try again
I0928 15:01:02.211146       1 azure_loadbalancer.go:135] ensure(test-azure-disk): FINISH - xxx.xxx.xxx.xxx

Is this related to persistent volume claims? Do we need to create the Azure storage class first?