netflix / titus-executor

Titus Executor is the container runtime/executor implementation for Titus

Virtual-Kubelet executor for Titus (ARCHIVED)

This is a historic archive of titus-executor.

---

Runs Titus containers using Docker.

Building

Prerequisites

You must have Docker 1.13+ installed on your system, and it must be running. The steps to install it can be found on the docker website. Once it is installed, you may have to run the following command, prior to logging out and back in again:

gpasswd -a $YOURUSER docker # Fill in $YOURUSER with your user name

You must have a Go 1.12 development environment set up. You can find more details on installing Go, on their website. You must also install the build-essential metapackage on Linux.

We recommend setting up a GOPATH, and checking out your code into that GOPATH. For example:

mkdir -p ${HOME}/go
## You add these lines to your .bashrc
export GOPATH=${HOME}/go
export PATH=${GOPATH}/bin:${PATH}
##

And then you can check out github.com/Netflix/titus-executor to ${GOPATH}/src/github.com/Netflix/titus-executor -- an easy way to do this is go run go get -u github.com/Netflix/titus-executor.

Initial setup steps

Ensure that your build environment (i.e. VM) has the following commands prior to building:

• make
• gcc

Building and testing

In order to build titus-executor, check out the project into your $GOPATH/src/github.com/Netflix, and run the following command:

sudo -E PATH=${PATH} make builder all

This will output a debian file at the path, which you can then install on your system: ./build/distributions/titus-executor_latest.deb

To only build the .deb, and not rebuild the builder image:

sudo -E PATH=${PATH} make build

Building without Docker

If you want to build a dpkg, without Docker, once the code is checked out, you can run the following:

make build-standalone

Linting

Linting is done via the golangci-lint package, which runs various linters.

To run lint checks:

# Lint all files:
make lint
# Run lint checks inside a docker container:
make validate-docker

Testing

Local Testing

You should be able to run "local" testing on your system. These are going to be tests that are primarily unit tests, and test for logical correctness, and not for correctness of interaction with system daemons:

make test-local

Tests will run locally, according to whatever platform you're on.

Standalone tests in a Docker container

Requires docker:

make test-standalone

# Disable running tests in parallel and change the test timeout (useful on slower systems):
TEST_FLAGS="-v -parallel 1" TEST_TIMEOUT=10m make test-standalone
# If you're iterating on tests and don't want to build a new executor .deb every time:
./hack/tests-with-dind.sh

Tests will run inside a Docker container and run a dedicated docker daemon as docker-in-docker.

AWS specific features (VPC integration, metadata service proxy, GPU, EFS, ...) are disabled during these tests.

Generated Code

There are places inside of the executor where we've checked in binaries, or prebuilt pieces of code. This may be considered harmful by some, but from the perspective of pragmatism, we have such code in:

In order to generate code you need go-bindata, which you can get via:

go get -u github.com/jteeuwen/go-bindata/...

vpc/bpf

There are two pieces of generated code:

• vpc/bpf/filter.o: This is compiled from vpc/bpf/filter.c. We ship this precompiled because it changes infrequently, and it requires LLVM, and the full kernel headers
• vpc/bpf/filter/filter.go: This is a go package generated using go-bindata based upon vpc/bpf/filter.o

In order to regenerate this data, if you edit vpc/bpf/filter.c, you can do so by running:

make -C vpc/bpf/

api/netflix/titus

If you update the dependency github.com/Netflix/titus-api-definitions/src/main/proto/netflix/titus, you will need to regenerate the Go proto definitions.

You must first install the protobuf toolchain, and the Go protobuf compiler. More documentation on that can be found in the protobuf repo.

Once you update, and sync the dependency, just run the following:

make clean-proto-defs protogen

Running

nvidia GPUs

To use with nvidia devices, you must have an OCI runtime installed that supports running OCI prestart hooks, so that the nvidia-container-runtime-hook can be run before container start. You'll need to add a container runtime as per the dockerd documentation. The executor looks for a runtime named oci-add-hooks by default, but the runtime can be configured via the titus.executor.nvidiaOciRuntime config option. You can use the nvidia-docker runtime, or the oci-add-hooks runtime if you're not comfortable running a patched version of runc.

LICENSE

Copyright (c) 2021 Netflix, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

titus-executor's Issues

Make Generic Cancel Launch Test

Make a runner-oriented test to see if we can cancel jobs while starting.

Replace the configuration system with Viper

Remove bind mounts

[Q] Can not compile titus-executor

Hi!
Thank you for releasing the source code for titus. I really want to try it.

I'm following your published instructions and you're failing me in the build step.

Attached is the log output
log.txt

There seems to be some problem with some go dependency (govendor and go-junir?)

| tee checkstyle-result.xml
go get github.com/kardianos/govendor
bash: govendor: no se encontró la orden
go get github.com/jstemmer/go-junit-report
bash: govendor: no se encontró la orden
govendor test -v -parallel 32 +local \
| tee /dev/stderr \
| tee test-local.log \
| go-junit-report > test.xml
bash: govendor: no se encontró la orden
bash: govendor: no se encontró la orden
bash: línea 3: go-junit-report: no se encontró la orden
make: *** [Makefile:57: test-local] Error 127

Thanks

Check if Digest Image Already Downloaded

Right now, the executor always forces us to go back to the registry. It would be nice to have a mode where it didn't go back to registry and checked if we already downloaded an image.

Add mechanism to GC "leaked" cgroups

Make it so that the launchguard can be disabled / bypassed based on protobuf

See: Netflix/titus-api-definitions#19

Remove launchguard

Configure the Circle CI builder to only build the builder image sometimes

Right now, the circle CI image builds the builder image always. It'd be good if this wasn't the case.

Replace our metrics usage with armon/go-metrics

Titus-executor build problems

Hi

I'm a master student researching container orchestration software stacks.
I'm having troubles building the titus-executor deb file.
After following instructions on the executor wiki
sudo -E PATH=${PATH} make builder all fails at the local-test check. I've included the test-local.log.

make build-standalone results in errors when executing gox commands:

tguvdamm@n104-05:~/go/src/github.com/Netflix/titus-executor$ make build-standalone
git submodule update --init --recursive
hack/builder/titus-executor-builder.sh
~/go/src/github.com/Netflix/titus-executor ~/go/src/github.com/Netflix/titus-executor
flag provided but not defined: -gcflags

The solution stated by logicbomb does not work for me, the same test fails.

So in the end, the $GOPATH/src/github.com/Netflix/titus-executor/build/distributions/titus-executor_latest.deb file does not exitst.
Any ideas as to why building fails?

Maybe some relevant extra info:
-I had to manually clone the titus-executor repo and tini submodule, go was denied access, even when elevated as sudo
-I had to install gox via apt
-system info: Ubuntu 18.04/amd64, the deployment page states Titus Agents are currently run on Trusty, but compiling the code should be fine on any platform I guess.

Thanks

Switch to Darion per Task

Try using Caddy + Systemd based injection? Maybe just some simple nsenter?

Add test around images with healthchecks

Use New go bindata

We should use the new go-bindata, from: go get -u github.com/go-bindata/go-bindata/..., which follows modern gofmt rules.