capgemini / kubeform

Form your :boat: Kubernetes :anchor: cluster anywhere using CoreOS, Terraform and Ansible

Kubeform

Deploy yourself a high-availability Kubernetes cluster, in minutes. Built on Terraform, CoreOS and Ansible.

Our recipes for bootstrapping HA Kubernetes clusters on any cloud or on-premise.

Includes the following -

• CoreOS as the base operating system
• Kubernetes (in HA) mode (leader election using Podmaster)
• SSL certs/security for Kubernetes cluster components
• Flannel for networking
• Kubernetes Dashboard
• Sky/KubeDNS

and optionally -

• Prometheus for cluster monitoring (coming soon!)
• Fluentd, elasticsearch for cluster logging
• Traefik as the ingress controller for the edge-routers. For configuring it to use letsencrypt you can edit this file.

See our Roadmap for future features and feel free to help us improve the project by contributing a Pull Request, or raise an issue if you run into trouble!

Getting started

Check out the instructions for provisioning on different clouds including:

• AWS
• Digitalocean
• Local Docker Compose

Demo

Check out this demo deploying and scaling the backends for the ingress-controller on AWS and DigitalOcean

Keep up to date...

Check out the Capgemini UK Engineering blog to find out more about the stuff we do!

kubeform's Issues

Setup kube/skydns by default in the cluster

See https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns

Protect docker with TLS certificates

See https://docs.docker.com/articles/https

Get kube-worker up properly and able to talk to the master

At the moment the service comes up but not yet properly. You cannot communicate back to the API server from the worker node. This is purely because that's as far as i've got testing for now.

This might be related to #11

The following error fills the logs -

Mar 15 13:19:49 k8s-worker-0 kubelet-wrapper[1483]: E0315 13:19:49.176393    1483 kubelet.go:1356] Failed creating a mirror pod "kube-proxy-k8s-worker-0_kube-system": Post http://46.101.30.214:8080/api/v1/namespaces/kube-system/pods: dial tcp 46.101.30.214:8080: connection refused
Mar 15 13:19:49 k8s-worker-0 kubelet-wrapper[1483]: E0315 13:19:49.176998    1483 kubelet.go:1361] Mirror pod not available

Seems like the API server isn't responding properly on it's publicly addressable URL - it only responds from localhost:8080 on the master node.

Add docs for addons/dns

• How to test (using busybox.yaml)
• How to use

Move flannel configuration out from cloud config to its own ansible role

For example see
https://github.com/Capgemini/apollo-kubernetes/blob/master/terraform/digitalocean/master-cloud-config.yml.tpl#L29-L49

Generate admin user credentials (service accounts) for http basic auth for the api server

We missed that out initially. See http://kubernetes.io/docs/getting-started-guides/scratch/#preparing-credentials (first bit)

See also http://kubernetes.io/docs/admin/authentication/

Create playbooks to handle a cluster upgrade

https://coreos.com/kubernetes/docs/latest/kubernetes-upgrade.html

Kubelet - Can't get ip address of node k8s-worker-0, so node addresses will be stale

Following error -

Can't get ip address of node k8s-worker-0, so node addresses will be stale: lookup k8s-worker-0: no such host

This probably because all we have in /etc/hosts is

127.0.0.1 k8s-worker-0

So we need to take care of DNS better or update /etc/hosts with more config entries. You'll see a similar error across all nodes if you run journalctl -xef -u kubelet.

Add Authorisation support to apiserver

Do this via Dex https://github.com/coreos/dex/tree/master/contrib/k8s
https://gist.github.com/yifan-gu/6fce1016a4cfe4c40d9c

hyperkube/coreos versions compatibly

The hyperkube entrypoint is /hyperkube in version 1.2

Seems to cause issues with kube-wrapper for our version of coreos

Create a 'vagrant' provider for the cluster setup

Should be multinode to simulate a real cluster as far as possible

Explore to support weave and consider sdn network pluggability

Opening this just for discussion #31

Getting started guide for DO

Create CLI

Proposal to create a CLI wrapper around creating/running/destroying kubeform clusters.
As an initial first pass this should support create and destroy.

We can potentially use this issue to hash out the details. Things to address -

• Support for config (YAML / command line argument / environment vars)?
• Which CLI libs to use (potentially https://github.com/codegangsta/cli)

Probably loads more stuff feel free to drop comments

Allow cluster monitoring addon to be deployed with the cluster

See https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-monitoring

Provide a load balancer across the master API server nodes

Should look something like this -

The worker then needs updated -

https://github.com/Capgemini/apollo-kubernetes/blob/master/roles/k8s-worker/templates/kube-proxy.yaml.j2#L14 Would then be modified to go through the load balancer rather than direct to a single master (SPOF at the moment)

Create some docs around helm

See #70

Setup CI builds for the project

Create a gh-pages docs site

Could do something similar to redux - use gitbook and publish via an npm script to the gh-pages branch (which will host on github.io)?

Create GCE terraform plan for k8s

Support ingress

Would be good to have some out-of-the box options for Kubernetes Ingress controllers.

Add docs for getting-started-guides/digitalocean

Ensure kubernetes api server certificates include correct dns_names

See https://coreos.com/kubernetes/docs/latest/openssl.html

pasting here

OpenSSL Config
This is a minimal openssl config which will be used when creating the api-server certificate. We need to create a configuration file since some of the options we need to use can’t be specified as flags. Create openssl.cnf on your local machine and replace the following values:

Replace ${K8S_SERVICE_IP}
Replace ${MASTER_HOST}
openssl.cnf

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
IP.1 = ${K8S_SERVICE_IP}
IP.2 = ${MASTER_HOST}
If deploying multiple master nodes in an HA configuration, you may need to add more TLS subjectAltNames (SANs). Proper configuration of SANs in each certificate depends on how worker nodes and kubectl users contact the master nodes: directly by IP address, via load balancer, or by resolving a DNS name.

Example:

DNS.5 = ${MASTER_DNS_NAME}
IP.3 = ${MASTER_IP}
IP.4 = ${MASTER_LOADBALANCER_IP}

Means we might need to set https://www.terraform.io/docs/providers/tls/r/self_signed_cert.html#dns_names

and https://www.terraform.io/docs/providers/tls/r/self_signed_cert.html#ip_addresses

when we create the certs

Upgrade to kubernetes 1.2

Issue to track upgrade to k8s 1.2. also related -
#27
#46

Provide an easy way to Install and configure kubectl for the cluster

Normally kubectl would be installed on the host (your machine) - We should make that an easy process and integrate with the certificate management that we are setting up for the cluster - see https://coreos.com/kubernetes/docs/latest/configure-kubectl.html

Move setup-network-environment.service to ansible

The below is current deployed by cloud config. Seems best to keep cloud config fairly light-touch and move this to ansible if we can to live in a 'common' role or somewhere near network related code (when we also move flannel).

name: setup-network-environment.service
      command: start
      content: |
        [Unit]
        Description=Setup Network Environment
        Documentation=https://github.com/kelseyhightower/setup-network-environment
        Requires=network-online.target
        After=network-online.target

        [Service]
        ExecStartPre=-/usr/bin/mkdir -p /opt/bin
        ExecStartPre=/usr/bin/curl -L -o /opt/bin/setup-network-environment -z /opt/bin/setup-network-environment https://github.com/kelseyhightower/setup-network-environment/releases/download/v1.0.0/setup-network-environment
        ExecStartPre=/usr/bin/chmod +x /opt/bin/setup-network-environment
        ExecStart=/opt/bin/setup-network-environment
        RemainAfterExit=yes
        Type=oneshot

Integrate an ansible module for kubernetes so we can easily create resources without CURL

This will allow us to create pods/rcs/etc... more easily

There is this one in the pipeline we might be able to re-use in the meantime
ansible/ansible-modules-extras#1229

might be others elsewhere - need to check properly.

Change flags for master on version 1.2

kubernetes/kubernetes#15108
On upgrade to 1.2 we need --register-node=true --register-schedulable=false so we remove noisy logs

Move ca / etcd cert generation to come from tf_tls module

Create a main README.md file for the project

refactor addons to be packaged and installed via helm charts

Add support for ubernetes-lite

Requires a 1.2 upgrade. See https://github.com/kubernetes/kubernetes/blob/release-1.2/docs/proposals/federation-lite.md

Validate / test the cluster provisioning

Maybe we can hook into the k8s validate cluster script to provide some validation ?

See https://github.com/kubernetes/kubernetes/blob/master/cluster/validate-cluster.sh

Allow private docker registry to be deployed with the cluster

See https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry

Investigate deis workflow integration

https://github.com/deis/workflow

Add configurable --cloud provider support for kubectl

This should come when we tackle gce/aws but needs to be configurable on/off in the template

Setup kubernetes dashboard by default in the cluster

See https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dashboard

setup + configure ELB for AWS kube master instances

hangover from #64

Configure docker startup options

Need to figure out the correct/best options (and make them configurable) - See http://kubernetes.io/docs/getting-started-guides/scratch

Add docs for addons/dashboard

Some docs required for the kubernetes dashboard

• how to access it
• firewall rules (if exposing externally)
• how to set up ingress for it etc..

Add CI step to publish to gitbook on merge to master

Could we do something like https://github.com/leipert/step-git-push / http://blog.wercker.com/2013/07/25/Using-wercker-to-publish-to-GitHub-pages.html to automate building the gh-pages branch only on merges to master?

We'd need to have npm available in the box (to get the gitbook cli). We should be able to override the box using the new wercker workflow pipelines

The alternative would be to use something else (travis?) to build the docs only

Support helm

Wonder if we should provide support (optionally?) for Helm https://github.com/kubernetes/helm

Create Azure terraform plan for k8s

kubelet on master displaying error about kube-system manifest

journalctl -xef -u kubelet (on any master node)

ile.go:123] Can't process config file "/etc/kubernetes/manifests/kube-system.yaml": /etc/kubernetes/manifests/kube-system.yaml: read 'apiVersion: v1
May 05 13:46:37 ip-10-0-1-60.eu-west-1.compute.internal kubelet-wrapper[20425]: kind: Namespace
May 05 13:46:37 ip-10-0-1-60.eu-west-1.compute.internal kubelet-wrapper[20425]: metadata:
May 05 13:46:37 ip-10-0-1-60.eu-west-1.compute.internal kubelet-wrapper[20425]: name: kube-system
May 05 13:46:37 ip-10-0-1-60.eu-west-1.compute.internal kubelet-wrapper[20425]: ', but couldn't parse as pod(invalid pod: &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:kube-system GenerateName: Namespace: SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[]} Spec:{Finalizers:[]} Status:{Phase:Active}}).
May 05 13:46:57 ip-10-0-1-60.eu-west-1.compute.internal kubelet-wrapper[20425]: I0505 12:46:57.077822```

Change usage of CURL to ansible uri module

we need to wait on Ansible 2.1 (so the dependency on httplib is removed for CoreOS) -

this code needs updated - https://github.com/Capgemini/kubeform/blob/master/roles/k8s-master/tasks/main.yml#L88

and we need to modify wercker.yml

http://docs.ansible.com/ansible/uri_module.html

Add docs for getting-started-guides/aws

Fix up AWS terraform scripts for k8s

These were copied over from apollo - but need a bit of rework and need to handle the certs correctly too here

Replace is_first_master with run_once

Replace is_first_master with run_once
https://github.com/Capgemini/kubeform/search?utf8=%E2%9C%93&q=is_first_master

Allow fluentd-elasticsearch addon to be deployed with the cluster

See https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

Add getting started guides section