vmware-archive / simple-k8s-test-env

For developers building and testing Kubernetes and core Kubernetes components

---

VMware has ended active development of this project, and its repository will no longer be updated.

---

Simple Kubernetes Test Environment

The Simple Kubernetes Test Enviornment (sk8) project is:

• For developers building and testing Kubernetes and core Kubernetes components
• Capable of deploying any version of Kubernetes (+1.10) on generic Linux distributions
• Designed to deploy single-node, multi-node, and even multi-control plane node clusters
• Able to deploy nodes on DHCP networks with support for both node FQDNs and IPv4 addresses
• A single, POSIX-compliant shell script, making it portable and customizable

Quick start

The quickest way to provision a Kubernetes cluster with sk8 is on vSphere using the OVA.

Getting started

• How does sk8 work?
• What does sk8 install?
• How to provision Kubernetes with sk8

How does sk8 work?

The sk8 project revolves around a single, POSIX-compliant shell script designed to be compatible with most Linux distributions. This model illustrates an example sk8 execution.

What does sk8 install?

A sk8-provisioned cluster passes the Kubernetes e2e conformance test suite because sk8 uses a well-known, standard set of components to the control plane and worker nodes:

How to provision Kubernetes with sk8

There are several ways to provision a Kubernetes cluster with sk8:

• Provision a multi-node cluster on vSphere with the sk8 OVA
• Provision a multi-node cluster on VMware Cloud (VMC) on AWS with the sk8 OVA
• Provision single-node and multi-node clusters with sk8 and Vagrant

Todo

• Better testing
• Better documentaton

License

Please the LICENSE file for information about this project's license.

simple-k8s-test-env's Issues

Any way to control Docker version?

After using sk8 to successfully deploy a K8s cluster on VMC (PhotonOS), the resulting version of Docker shows as 17.06-ce.

It was brought up to me today that recommended version of docker is 18.09. Is there a way to have sk8 install a newer version? or is this a limitation of what is packaged for Photon?

I have no indication at this point that anything is wrong with 17.06-ce. Just something I wanted to bring up.

e2e: Failing "should rollback without unnecessary restarts [Conformance]" on ci/latest

The sk8 e2e process, which relies on Sonobuoy, no longer passes the e2e conformance test [sig-apps] Daemon set [Serial] [It] should rollback without unnecessary restarts [Conformance]:

1. kubernetes/kubernetes#74853
2. heptio/kube-conformance#33

e2e: Failing "Should be able to support the 1.10 Sample API Server using the current Aggregator [Conformance]" on ci/latest

The sk8 e2e process, which relies on Sonobuoy, no longer passes the e2e conformance test [sig-api-machinery] Aggregator [It] Should be able to support the 1.10 Sample API Server using the current Aggregator [Conformance].

The initial issue is that sk8 does not currently does not include the requestheader-client-ca-file flag when configuring the API server. A PR will be opened shortly that has a patch to rectify this problem. However, the PR doesn't completely solve the issue as this test still fails, even with the flag.

Please see this gist for a few report of the problem, including logs.

Stuck deploying on vsphere via ova

I'm following the provision-on-vsphere-with-ova doc and imported the OVA like this:

VC=vcenter
[email protected]
PASS=pass
DC1=my_domain_controller1
DC2=my_domain_controller2
OVA=/path/to/sk8-photon.ova
DC=datacenter
CLUSTER=cluster
HOST=esx
DS=datastore
NAME=k8s
OVA_NET=sddc-cgw-network-3
TA_NET=my-network

ovftool --acceptAllEulas \
        --allowExtraConfig \
        --datastore=$DS \
        --disableVerification \
        --diskMode=thin \
        --machineOutput \
        --noSSLVerify \
        --name=$NAME \
        --net:$OVA_NET=$TA_NET \
        --powerOn \
        --prop:CLOUD_PROVIDER_TYPE=In-tree \
        --prop:K8S_VERSION=release/stable \
        --prop:NETWORK_DNS_1=$DC1 \
        --prop:NETWORK_DNS_2=$DC2 \
        --prop:NUM_BOTH=1 \
        --prop:NUM_CONTROLLERS=2 \
        --prop:NUM_NODES=3 \
        --prop:VSPHERE_PASSWORD=$PASS \
        --prop:VSPHERE_SERVER=$VC \
        --prop:VSPHERE_USER=$USER \
        --skipManifestCheck \
        $OVA \
        vi://$USER:$PASS@$VC/$DC/host/$CLUSTER/$HOST

after some time, the 3 VMs are up and running. But when I run this:

curl -sSL http://bit.ly/sk8-local | sh -s -- MYUUID

I get this:

getting cluster information                                        
  * id                          govc: specify an ESX or vCenter URL

Please help me find the little detail I missed.

• Do I need to set another OVF property? Is one incorrect?
• Is In-tree the right CLOUD_PROVIDER_TYPE for my on-prem vcenter/esx setup?
  • BTW, I'm on vcenter 6.7.0.20000 and ESXi 6.7.0 10302608

Thanks.

kube-apiserver fails to start v1.14.1: unrecognized

When I deploy with the latest stable release, version v1.14.1. I see that the kube-apiserver is failing to start. It does not recognize the Initializers admissions plugin. Looks the the version test is failing. Maybe '1' < 'a'? I kicked off v1.12.7 and it works fine.

if [ "${_ge_v1_14_alpha_1}" -ge "0" ]; then
APISERVER_OPTS_ENABLE_ADMISSION_PLUGINS="${APISERVER_OPTS_ENABLE_ADMISSION_PLUGINS:-NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota}"
else
APISERVER_OPTS_ENABLE_ADMISSION_PLUGINS="${APISERVER_OPTS_ENABLE_ADMISSION_PLUGINS:-Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota}"
fi

set -o pipefail in user script is not understood by sh

The notes in c01 suggest piping the script into sh. On my version of Linux, sh doesn't understand set -o pipefail, so it only works if piped to bash.

Not a big deal, but maybe worth considering bash as an alternative

sk8 deployment to vSphere env fails with multiple datacenters

After powering on sk8, I found that there were no Controller/Worker nodes deployment. I logged into the c01 and looking at /var/log/sk8/vsphere.log, it looks like the code assumes there's only a single vSphere datacenter

govc: default datacenter resolves to multiple instances, please specify
FATAL [1548905820] failed to get VM's info

Wondering if a quick solution would be to simply look up the currently deployed VM and then walk back up to the parent Datacenter and use that as inventory path?

Here's what my vSphere Inventory looks like in case that helps:

expired root password

When deploying the appliances, the root password is showing as expired and it forces a password change... apparently this is also breaking the kube deployment as well:
/var/lib/sk8/sk8.sh: line 4336: warning: command substitution: ignored null byte in input
kubernetes/server/bin/kube-apiserver
kubernetes/server/bin/kube-scheduler
kubernetes/server/bin/kube-controller-manager
INFO [1570030497] inflated /var/lib/kubernetes/install/remote/kubernetes-server-linux-amd64.tar.gz
INFO [1570030497] already downloaded nginx
INFO [1570030497] already downloaded coredns
You are required to change your password immediately (password expired)
groupadd: PAM: Authentication token is no longer valid; new one required
FATAL [1570030497] failed to create the k8s-admin group