cybozu-go / sabakan

A versatile network boot server for large data centers

Sabakan

Sabakan is a versatile network boot server designed for large on-premise data centers. Currently, it is made only for Flatcar Container Linux.

Project Status: GA (General Availability)

Features

• High availability

  High availability of sabakan is just as easy as running multiple sabakan servers.

  Sabakan data are stored and shared in etcd. For example, DHCP lease information are shared between sabakan instances to avoid conflicts.

• Machine inventory with IPAM (IP address management)

  Sabakan keeps an inventory of machines in a data center. Their IP addresses are automatically assigned by sabakan.

• DHCP service

  Sabakan provides DHCP service that supports UEFI HTTP Boot and iPXE HTTP Boot. It also supports DHCP relay request to make DHCP service highly available.

• HTTP service (network file server)

  Sabakan provides HTTP service for network boot clients. Users can upload any kind of files other than OS images to sabakan. Clients can download them to initialize the system after boot.

• Template system for Ignition

  Ignition is a boot provisioning system for Flatcar Container Linux. Ignition configuration is not friendly for operators as it is written in a plain JSON.

  Sabakan provides a friendly and super versatile template system for Ignition configurations. For each client machine, sabakan renders Ignition configuration from templates.

• Life-cycle management

  Machines in the inventory has a life-cycle status. The status can be changed through REST API. Users can build an automatic status controller to mark machines as unhealthy, unreachable, retiring, or retired.

• Disk encryption support

  To help implementing full disk encryption on client machines, sabakan accepts and stores encrypted disk encryption keys. The key can be downloaded in the next boot to decrypt disks.

  sabakan-cryptsetup is a tool for clients to encrypt disks; the tool generates a disk encryption key, encrypts it, and sends the encrypted key to sabakan. In the next boot, it downloads the encrypted key from sabakan, decrypts it, then uses it to decrypt disks.

• Audit logs

  To track problems and life-cycle events, sabakan keeps operation logs within its etcd storage.

Programs

This repository contains these programs:

• sabakan: the network service to manage servers.
• sabactl: CLI tool for sabakan.
• sabakan-cryptsetup: a utility to encrypt a block device using dm-crypt.

To see their usage, run them with -h option.

Documentation

docs directory contains tutorials and specifications.

Read getting started first.

Examples

mtest/ directory contains a set of utilities to setup sabakan on Ubuntu virtual machines.

testadata/ directory contains a sample Ignition template.

An example of production usage can be found in github.com/cybozu-go/neco. The repository bootstraps a full data center system using etcd, vault, sabakan, and many other tools.

Usage

Run sabakan with docker

# create directory to store OS images
$ sudo mkdir -p /var/lib/sabakan

# create server certificate
$ sudo mkdir -p /etc/sabakan
$ make setup-cfssl
$ cd e2e/certs && ./gencerts.sh
$ cd ../..
$ sudo cp e2e/output/certs/server.crt /etc/sabakan/server.crt
$ sudo cp e2e/output/certs/server.key.insecure /etc/sabakan/server.key

# -advertise-url is the canonical URL of this sabakan.
$ docker run -d --read-only --cap-drop ALL --cap-add NET_BIND_SERVICE \
    --network host --name sabakan \
    --mount type=bind,source=/var/lib/sabakan,target=/var/lib/sabakan \
    --mount type=bind,source=/etc/sabakan,target=/etc/sabakan \
    ghcr.io/cybozu-go/sabakan:3.1 \
    -etcd-endpoints http://foo.bar:2379,http://zot.bar:2379 \
    -advertise-url http://12.34.56.78:10080 \
    -advertise-url-https http://12.34.56.78:10443

License

Sabakan is licensed under the Apache License, Version 2.0.

Docker images

Docker images are available on ghcr.io

sabakan's Issues

Sabakan has duplicated assets' URLs

Describe the bug
A clear and concise description of what the bug is.

$ sabactl assets info lvmd
{
  "name": "lvmd",
  "id": "1183",
  "content-type": "application/octet-stream",
  "date": "2021-04-09T01:34:26.558150602Z",
  "size": 26234646,
  "sha256": "6140daf5e3d102c5abc751bcf39596afe34a78da9426530fbb8a4c9844b14840",
  "urls": [
    "http://10.69.0.195:10080/api/v1/assets/lvmd",
    "http://10.69.1.131:10080/api/v1/assets/lvmd",
    "http://10.69.0.3:10080/api/v1/assets/lvmd",
    "http://10.69.2.67:10080/api/v1/assets/lvmd",
    "http://10.69.3.3:10080/api/v1/assets/lvmd",
    "http://10.69.3.3:10080/api/v1/assets/lvmd",
    "http://10.69.2.67:10080/api/v1/assets/lvmd",
    "http://10.69.1.131:10080/api/v1/assets/lvmd",
    "http://10.69.0.195:10080/api/v1/assets/lvmd",
    "http://10.69.0.3:10080/api/v1/assets/lvmd"
  ],
  "exists": true,
  "options": {}
}

Environments

• Version: 2.13.2
• OS:

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

Consistent Machine ID

In order to keep a persistent /etc/machine-id during the lifetime of a Machine,
sabakan should generate a unique 32-character hexadecimal (= 16-byte) value.

That value will be referenced from Ignition templates to set machine-id.

Related: cybozu-go/cke#64

dhcp renewal is not working

Describe the bug

When DHCP server receives a renwal request, the lease will not be extended.

v is not a pointer type.

sabakan/models/etcd/dhcp.go

Line 139 in 4bad63c

v.LeaseUntil = leaseUntil

Environments

• Version: v2.9.1

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

[KAIZEN] Improve API and command-line for labels

What

API to add labels is not well designed as it uses PUT for
the entire set of labels of a machine, but in fact it works as
adding new labels instead of replacing existing labels.

Furthermore, sabactl does not provide subcommands for labels.

Improve these bits.

How

Add a REST API PUT /api/v1/labels/<serial>/<label> to add/replace a label for a machine.

Note that DELETE /api/v1/labels/<serial>/<label> already exists. We don't need a REST API GET /api/v1/labels/<serial>/<label> to get a label value of a machine because this can be achieved with GET /api/v1/machines. GET does not need a server-side atomic operation.

Remove a REST API PUT /api/v1/labels/<serial>. It is not used anywhere.

Add sabactl machines *-label subcommands to manipulate labels.

• sabactl machines get-label <serial> <label> => use sabactl machines get
• sabactl machines set-label <serial> <label> <value>
• sabactl machines remove-label <serial> <label>

Checklist

• Finish implentation of the issue
• Test all functions
• Have enough logs to trace activities
• Notify developers of necessary actions

Opt in to Go modules

and run go vet in CI.

Support Ignition v3

What

From the Flatcar Container Linux 3.1.85.0.0, Ignition v3 is supported.
https://www.flatcar.org/docs/latest/provisioning/ignition/specification/#ignition-v3

How

Describe how to address the issue.

Checklist

• Finish implentation of the issue
• Test all functions
• Have enough logs to trace activities
• Notify developers of necessary actions

When the machine information is deleted, the metrics continue to remain.

What

After running sabactl machines remove ${SERIAL} to remove machine information, status and other metrics about ${SERIAL} continued to remain.
I did confirm that those metrics disappeared by restarting sabakan.

How

• Deletes metrics when machine information is deleted.

Checklist

• Finish implentation of the issue
• Test all functions
• Have enough logs to trace activities
• Notify developers of necessary actions

Prepare document for Maneki which explains how to use IngressRoute

What

To expose apps deployed by Maneki to the bastion network or the Internet, we need to explain how to use IngressRoute.

How

• Prepare a document which describes:
  • What kind of LoadBalancer services are deployed on the cluster.
  • How to expose apps to the Internet with IngressRoute.
  • How to expose apps to the bastion network with IngressRoute.

Checklist

• Finish implementation of the issue
• Test all functions
• Have enough logs to trace activities
• Notify developers of necessary actions

dummy for add a new label and all Neco related repos

Examples set auto fields but those values are overridden by sabakan

Describe the bug

The examples at:

https://github.com/cybozu-go/sabakan/blob/main/docs/getting_started.md#register-machines
https://github.com/cybozu-go/sabakan/blob/c77d1989472bc72ceb47b4b1620152a6b3e455fb/docs/machine.md#machine-struct

Set auto fields like bmc.ipv4 but those are overridden by sabakan, so I think those fields should be removed from the examples.

Also the documentation should explicitly mention that auto fields are overridden and must not be set by the user. I think the Values for auto fields are filled by sabakan at registration. phrase is not explicit enough, because its ambiguous whether they can/should be set afterwards.

Implement the feature to set kernel-params for each machine type.

What

In the current implementation, setting kernel-params applies to all machines.

How

Implement the feature to set kernel-params for each machine type.

Checklist

• Finish implentation of the issue
• Test all functions
• Have enough logs to trace activities
• Notify developers of necessary actions

Enable sabakan-cryptsetup test

What

sabakan-cryptsetup test are skipped in mtest.
I want to enable the test.
https://github.com/cybozu-go/sabakan/blob/v2.13.2/mtest/netboot_test.go#L98-L99

ref: https://github.com/cybozu-go/sabakan/pull/194/files

How

Describe how to address the issue.

Checklist

• Finish implentation of the issue
• Test all functions
• Have enough logs to trace activities
• Notify developers of necessary actions

Error: Bad Request: invalid request: labels contain invalid character

Describe the bug

While trying to follow the Getting Started / Register Machines procedure at https://github.com/cybozu-go/sabakan/blob/main/docs/getting_started.md#register-machines, it fails with:

Error: Bad Request: invalid request: labels contain invalid character

This is due to the space character in the labels.product machine definition:

[
  {
    "labels": {
      "product": "Dell R640"
    },

I think the code or documentation should be updated to prevent this error.

PS: there's a space at https://github.com/cybozu-go/sabakan/blob/c77d1989472bc72ceb47b4b1620152a6b3e455fb/docs/machine.md#machine-struct too

nilerr: internal error: nil Pkg importing "github.com/onsi/gomega/types" from "github.com/onsi/gomega/internal/assertion"

Hi.

Caught such an error:

$ make
test -z "$(gofmt -s -l . | grep -v '^vendor' | tee /dev/stderr)"
test -z "$(golint $(go list ./... | grep -v /vendor/) | grep -v '/mtest/.*: should not use dot imports' | tee /dev/stderr)"
test -z "$(nilerr ./... 2>&1 | tee /dev/stderr)"
nilerr: internal error: nil Pkg importing "github.com/onsi/gomega/types" from "github.com/onsi/gomega/internal/assertion"
make: *** [Makefile:26: test] Error 1

$ go version
go version go1.15.2 gollvm LLVM 12.0.0git linux/amd64

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/oceanfish81/.cache/go-build"
GOENV="/home/oceanfish81/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/oceanfish81/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/oceanfish81/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/oceanfish81/gollvm_dist"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/oceanfish81/gollvm_dist/tools"
GCCGO="/home/oceanfish81/gollvm_dist/bin/llvm-goc"
AR="ar"
CC="/usr/bin/clang"
CXX="/usr/bin/clang++"
CGO_ENABLED="1"
GOMOD="/home/oceanfish81/sabakan/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build032338975=/tmp/go-build -gno-record-gcc-switches -funwind-tables"

CC @tenntenn