isabella232 / vmkite

An agent that listens for buildkite jobs and spawns ephemeral VMWare environments in a vSphere cluster

vmkite

Provides a daemon that listens for builds on Buildkite and spawns ephemeral virtual machines running Buildkite Agent on a VMWare vSphere cluster. After the build has finished the instance is destroyed. This allows for a repeatable environment for testing.

Requires a vSphere Datastore with the base virtual machine image in a .vmx format. Currently MacStadium provides these for you as part of your private cluster, contact us for the location to use.

Running

Either install with go get github.com/macstadium/vmkite or install one of the releases for your platform.

vmkite run \
  --buildkite-agent-token=BUILDKITE-AGENT-TOKEN \
  --buildkite-api-token=BUILDKITE-API-TOKEN \
  --buildkite-org=BUILDKITE-ORG \
  --target-datastore=TARGET-DATASTORE \
  --source-datastore=SOURCE-DATASTORE \
  --vm-cluster-path=VM-CLUSTER-PATH \
  --vm-network-label=VM-NETWORK-LABEL \
  --vm-memory-mb=VM-MEMORY-MB \
  --vm-num-cpus=VM-NUM-CPUS \
  --vm-num-cores-per-socket=VM-NUM-CORES-PER-SOCKET \
  --vsphere-host=VSPHERE_HOST \
  --vsphere-user=VSPHERE_USER \
  --vsphere-pass=VSPHERE_PASS \
  --vsphere-insecure=false

Strategy

• Assume base images / VMs / templates loaded into VMware cluster.
• Poll Buildkite API for jobs matching vmkite-name=X where X is a known base/template.
• Check for available slots based on X virtual machines per host.
• Create VM with independent non-persistent disk from base image.
• VM launches Buildkite Agent with vmkite-name=X metadata.
• After a job, the VM shuts itself down.

Notes

Initially derived from makemac.go by Brad Fitzpatrick for golang/build is a rough-draft quality tool with similar objectives for building the Go codebase on macOS VMs. It wraps the govc CLI interface of the govmomi VMware Go library. It has a few code review comments.

makemac.go code review notes suggest booting from a template is slow, and that creating new VMs from scratch and attaching a frozen disk image is fastest.

govmomi is the official Go library for the VMware vSphere API. It has a CLI interface called govc.

go-buildkite is the Buildkite API client for Go.