kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management

kOps - Kubernetes Operations

The easiest way to get a production grade Kubernetes cluster up and running.

What is kOps?

We like to think of it as kubectl for clusters.

kops will not only help you create, destroy, upgrade and maintain production-grade, highly available, Kubernetes cluster, but it will also provision the necessary cloud infrastructure.

AWS (Amazon Web Services) and GCE (Google Cloud Platform) are currently officially supported, with DigitalOcean, Hetzner and OpenStack in beta support, and Azure in alpha.

Can I see it in action?

Installing and launching a Kubernetes cluster hosted on AWS, GCE, DigitalOcean, Hetzner, OpenStack, Azure

See Getting Started

Documentation

Documentation is in the /docs directory, and can be seen at kops.sigs.k8s.io.

Releases and kubernetes Release Compatibility

See Releases and versioning

Getting Involved and Contributing

See Contributing

Office Hours

kOps maintainers set aside one hour every other week for public office hours. This time is used to gather with community members interested in kOps. This session is open to both developers and users.

We do maintain an agenda and stick to it as much as possible. If you want to hold the floor, put your item in this doc. Bullet/note form is fine. Even if your topic gets in late, we do our best to cover it.

For more information about the office hours and how to join, see Office Hours

kops's Issues

Avoid `no buildable go source files` warning

From a clean repo, go get -d k8s.io/kops still warns about no buildable go files (warning seems harmless though). I think it's just a matter of putting an empty go file in the root.

Brought up a cluster on AWS with kops and cannot attach ELB PersistentVolume to Pods

Pod fails to create with

FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod ...

Every time I try to mount a pod with a elb attached.

Kops is using kubernetes 1.3.0

Any hints?

kubelet not configured to autostart on reboot

A reboot of a kops created instance doesn't seem to have the kubelet setup to autostart.

We should move the delete cluster output to the table helper

And also sort by type, then id, then name

User report: no EBS volumes on nodes

kops didn't create an EBS volume for the nodes, instead relying on the default 8gb volume to store data.

It would be nice if:

• It created an EBS volume
• It was easy to specify the desired size of the EBS volume

Running `kops import` against cluster with custom subnet will fail

Description:

Attempted to run the kops import command against a cluster created via kube-up.sh that was given an existing subnet_id during creation. kops import fails with the following:

I0706 09:38:30.361037   62744 import_cluster.go:78] Found master: "i-xxxxxxx"
F0706 09:38:30.660197   62744 import_cluster.go:26] cannot find subnet "subnet-xxxxxx"

This is because kube-up will not tag an existing subnet with the KubernetesCluster tag if you give it a custom subnet_id on creation.

Resolution

Manually tagging the subnet with KubernetesCluster tag, then running kops import command will resolve the issue.

We should validate that names are valid DNS names

• all lower-case
• "looks like" a domain - i.e. has dots etc
• other rules around valid DNS names

kops get cluster could be faster

We're reading more files than we need to.

instance type not handled: "r3.2xlarge"

W0706 15:09:14.248996 18462 executor.go:86] error running task "launchConfiguration/nodes.prod.kubernetes.": instance type not handled: "r3.2xlarge"

kops unable to determine which zone to use if public and private zones are present with the same name

In AWS, If there are two DNS zones with the same name, one public, one private, kops is unable to determine which zone is associated with the VPC.

Maybe an optional flag for zoneid would be good?

kops rolling-update should not require a region

It should work with a cluster-spec from the state store.

Heapster dies after being created

After using wget to save upup/addons/monitoring-standalone/heapster-controller.yaml and using kubectl create -f heapster-controller.yaml, Heapster will run for a while and then disappear without any errors from kubectl logs.

I ssh'd into the master node and noticed that there wasn't an /etc/kubernetes/addons/cluster-monitoring dir before, during, or after doing the above steps.

Would be safest to delete the /etc/kubernetes/manifests before mounting disk

We have a symlink from /etc/kubernetes/manifests to a manifest on the volume, to avoid the race in #73.

But once the mount the volume, it is possible that a different manifest exists, and the symlink then resolves.

It would be safer to delete the symlink before mounting the volume. Another option would be to use a UUID in the mount path.

I'm not sure this actually matters much, because the manifest shouldn't be changing anyway. But we should probably fix to prevent future issues.

We can remove glide instructions

We've embedded the vendor repo now...

Deleting non-existent cluster gives bad error message

> kops delete cluster --name does.not.exist  --yes
F0703 11:11:37.172050    1503 delete_cluster.go:35] sanity check failed: cluster name mismatch

Don't default to HA master

My thoughts on slack:

I recommend kops create cluster ... --zones us-east-1b,us-east-1c,us-east-1d --master-zones us-east-1b to get multizone without HA master for now
Unless you really want HA master. And I'm probably going to change the default so it doesn't push people that aren't expecting HA master to HA master
(single zone master is still relatively available. HA master is slower and not as well tested/known.)

Need a command to print all the specs

This will be useful for diagnostics / support. Not just the cluster spec, but also the instancegroups.

Write up overview doc of fi

Explaining what a task is, the lifecycle, how cloudup & nodeup build on it etc

should kops delete cluster remove DNS entries?

kops delete cluster does a really good job of cleaning up. One outlier seems to be DNS entries.

User report: error computing key fingerprint

Seems to be a problem in computeAWSKeyFingerprint; but the output is mangled so it isn't entirely clear.

Expose instancegroups through CLI

We have instancegroups, but they aren't currently exposed

kops should return better help message

when I run kops -h, all the flags I want to use (except -v) are not listed.

kops -h should list all the flags (like --cloud --zones, --master-zones, --dns-zone etc)

Our IAM policies do not work in cn-north-1

A client error (NoSuchEntity) occurred when calling the GetInstanceProfile operation: Instance Profile kubernetes-master cannot be found.
Creating master IAM profile: kubernetes-master
Creating IAM role: kubernetes-master

A client error (MalformedPolicyDocument) occurred when calling the CreateRole operation: Invalid principal in policy: "SERVICE":"ec2.amazonaws.com"

I believe the IAM service name should be ec2.amazonaws.com.cn

need to use VPC internal DNS

It would be nice to be able to use aws private DNS for the cluster name

Wait after creating IamInstanceProfile

IAM instance profile creation seems to be eventually consistent / async, so we usually get an annoying warning:

W0705 12:37:39.249095   10306 executor.go:86] error running task "launchConfiguration/nodes.redact": error creating AutoscalingLaunchConfiguration: ValidationError: Invalid IamInstanceProfile: nodes.jul5h.awsdata.com
        status code: 400, request id: e07c08e6-42ce-11e6-893f-a191f7d89793
W0705 12:37:39.250229   10306 executor.go:86] error running task "launchConfiguration/master-us-east-1b.masters.redact": error creating AutoscalingLaunchConfiguration: ValidationError: You are not authorized to perform this operation.
        status code: 400, request id: e17276d0-42ce-11e6-88b4-173f86f21f4f
I0705 12:37:39.250275   10306 executor.go:70] Tasks: 47 done / 51 total; 2 can run

The problem goes away on the next automatic retry, but it is off-putting

nodeup tar is not compiled in docker

So it relies on the building machine's OS. We should build it in docker, like we do our other components.

Import Cluster not working with HA master

Currently I believe that Import Cluster is not working with HA masters.
Command: kops import cluster --v=8 --region=us-west-2 --name=${NAME} --state=${KOPS_STATE_STORE}
Output:
I0706 12:06:36.953211 71665 context.go:131] Found bucket "redacted" in region "us-west-2" I0706 12:06:37.689813 71665 s3fs.go:169] Listed files in s3://redacted/redacted/pki/issued/ca: [s3://redacted/redacted/pki/issued/ca/6303889741236192579705609245.crt] I0706 12:06:37.689862 71665 s3fs.go:120] Reading file "s3://redacted/redacted/pki/issued/ca/6303889741236192579705609245.crt" I0706 12:06:37.764648 71665 ca.go:288] Parsing pem block: "CERTIFICATE" I0706 12:06:37.815254 71665 s3fs.go:169] Listed files in s3://redacted/redacted/pki/private/ca: [s3://redacted/redacted/pki/private/ca/6303889741236192579705609245.key] I0706 12:06:37.815298 71665 s3fs.go:120] Reading file "s3://redacted/redacted/pki/private/ca/6303889741236192579705609245.key" I0706 12:06:37.888631 71665 ca.go:306] Parsing pem block: "RSA PRIVATE KEY" I0706 12:06:37.890916 71665 import_cluster.go:499] Querying EC2 instances F0706 12:06:38.134138 71665 import_cluster.go:26] could not find master node

kops: reenable GCE

We disabled GCE because it was lagging behind AWS. Get it back up to par!

kops rolling-update should do a real rolling-update

Right now it is a hacky timing-based loop.

Ideally we would wait for the new node to be registered.

kops should accept the cluster name as a positional argument

i.e. kops delete cluster mycluster should work, not just kops delete cluster --name mycluster

Error after running in --dryrun mode in existing VPC

After running: `$GOPATH/bin/kops create cluster --dryrun --cloud=aws --master-zones=us-west-1b --zones=us-west-1b,us-west-1c --name=${MYZONE} --vpc=vpc-123456 --network-cidr=10.0.0.0/16``

Running a second time an error is returned:
E0702 22:36:57.274257 64548 create_cluster.go:200] Cannot change master-zones from the CLI

This may also be due to the master-zone override. Deleting the state from the s3 bucket works around this issue.

Switch to dns-controller

We're using my kope aws-controller, I have a PR for the dns-controller (cleaned up code, moving into this tree) but it is currently blocked on kubernetes/kubernetes#28477

The dns-controller is also the biggest blocker for GCE :-)

SSL/TLS generation for API

As mentioned in the Slack Channel, due to the fact that Kops encourages you to build a DNS name for your server, some sort of certificate management and generation should be included. Let's Encrypt would be a great option to allow free certificates to any domain that is used.

Let me know if there is any way for me to help work on this.

Allow specifying dns-name separately?

Maybe we should allow the ClusterName to be different from the DNS name. Users will likely want to specify a shorter name, for example. And this means we're less bound to DNS at the schema level.

On the flip side, it means cluster names are no longer globally unique.

Spot price for minions

Hi!
It'll be great to create node-spot-price option. Or maybe node-spot-prices (one price for all zones or separate prices per zone?),
Cheers,
Pawel

network-cidr is easily confused with subnet-cidr

Maybe we should support vpc-cidr as well / instead

User report: kops chose m4.xlarge nodes

Not clear why at the moment, but the cluster came up with m4.xlarge nodes

message about duplicate device mapping

I think this is with t2.micro

W0705 16:29:37.823631 31304 launchconfiguration.go:137] not attach ephemeral device - found duplicate device mapping: "/dev/sdc"

We should validate the cluster after edit, before writing the file

Otherwise it is possible to get "stuck" in a bad configuration

Handling addons for HA master

As mentioned in Slack, I believe some work needs to be done with handling addons when it comes to having HA masters across different Zones.

Route53 zone creation is broken

Not the worst thing in the world, because typically users want to reuse an existing hosted zone (and our --zone-name logic is a little wonky). But if we're going to do it, we should do it right!

W0705 15:08:45.733037     888 executor.go:86] error running task "dnsZone/domain.com": error creating DNS HostedZone: InvalidParameter: 1 validation error(s) found.
- missing required field, CreateHostedZoneInput.CallerReference.

error running `kops addons`

From slack:

Strange bug with Kops: when I run kops addons get I get this error: F0705 15:43:47.931726 60722 addons_get.go:33] error adding SSH private key "error parsing key file "/Users/redacted/.ssh/id_rsa": asn1: structure error: length too large": %!v(MISSING)

User report: unable to use custom public key

From slack:

I was not able to select a custom publickey for the cluster for some reason
it would only let me use default ~/.ssh/id_rsa.pub
when I tried another keys it crashed

Maybe a non-RSA key?

Being able to pass docker_extra_opts to nodes

For being able to pass --insecure-registry for example

Possible race condition (?) when restarting master

If we restart the master, and the disks were mounted previously, the etcd manifest might be in /etc/kubernetes/manifests. But after a restart we might no longer be able to mount the etcd volumes (someone else may have got them), and we might also start etcd before mounting the disks.

kops create cluster should fail if --name already exists

Steps to reproduce:

Create cluster with:
${GOPATH}/bin/kops create cluster --v=0 --cloud=aws --zones=us-west-2c --name=${NAME}

Accidentally run:
${GOPATH}/bin/kops create cluster --v=0 --cloud=aws --zones=us-east-1c --name=${NAME}
(same name, different region)

This will break the cluster config. kops should check that --name doesn't already exist in the state_store before committing changes to the config.

Spot price for master

Hi!
I know - that's controversial :-) But not everyone are using Kubernetes on production, some of users want to test it, then spot instances should be ok. Imho it must be combined with HA option (for example 3 masters across 3 AZ).
Cheers,
Pawel

Do we need a 'update' command

We have kops edit which lets a user edit a cluster, but then the user is expected to re-run kops create cluster. That is a confusing command to run. We should probably borrow from kubectl, and have kubectl edit and kubectl apply also rerun the configuration.

docs: we should document how to choose a name

It isn't 100% clear how to choose a cluster name. Give some suggestions!

(and also document --zone-name more)

I like to do things like --name=kubernetes.sub.domain.com --zone-name=sub.domain.com, or even --name=dev-k8s.sub.domain.com, or dev.k8s.sub.domain.com.

Given a pool of Elastic IPs, provision them to Ingress Controllers

Basic use case: Some customers require whitelisting IP Addresses in order to access a service. This precludes the ability to leverage ELBs or similar.

Suggested implementation: Admin provides a pre-existing list of EIP Allocations, The controller should then label the nodes which receive these EIPs and then associate the EIPs with those nodes.

Then the admins/developers can target jobs at those labels.

Ideally, several pools of EIPs should be possible, with a unique label per EIP Pool.