GithubHelp home page GithubHelp logo

samuelkarp / runj Goto Github PK

View Code? Open in Web Editor NEW
601.0 41.0 32.0 544 KB

runj is an experimental, proof-of-concept OCI-compatible runtime for FreeBSD jails.

License: Other

Makefile 0.94% Go 98.30% Shell 0.14% Smarty 0.62%
freebsd containers oci open-containers jails freebsd-jail

runj's Introduction

runj

runj is an experimental, proof-of-concept OCI-compatible runtime for FreeBSD jails.

Important: runj is a proof-of-concept and the implementation has not been evaluated for its security. Do not use runj on a production system. Do not run workloads inside runj that rely on a secure configuration. This is a personal project, not backed by the author's employer.

Status

Build Status

runj is in early development and is functional, but has very limited features.

runj currently supports the following parts of the OCI runtime spec:

  • Commands
    • Create
    • Delete
    • Start
    • State
    • Kill
  • Config
    • Root path
    • Process args
    • Process environment
    • Process terminal
    • Hostname
    • Mounts
    • Hooks
      • CreateRuntime
      • Poststop

runj also supports the following experimental FreeBSD-specific extensions to the OCI runtime spec:

  • Config
    • IPv4 mode
    • IPv4 addresses

Getting started

OCI bundle

To run a jail with runj, you must prepare an OCI bundle. Bundles consist of a root filesystem and a JSON-formatted configuration file called config.json.

Experimental FreeBSD-specific extensions may be added directly to the config.json if desired, or may optionally be added to a runj-specific file located in the bundle directory called runj.ext.json.

Root filesystem

The root filesystem can consist either of a regular FreeBSD userland or a reduced set of FreeBSD-compatible programs. For experimentation, statically-linked programs from /recovery may be copied into your bundle. You can obtain a regular FreeBSD userland suitable for use with runj from http://ftp.freebsd.org/pub/FreeBSD/releases/$ARCH/$VERSION/base.txz (where $ARCH and $VERSION are replaced by your architecture and desired version respectively). Several demo convenience commands have been provided in runj to assist in experimentation; you can use runj demo download to retrieve a working root filesystem from the FreeBSD website.

Config

runj supports a limited number of configuration parameters for jails. The OCI runtime spec does not currently include support for FreeBSD, however runj adds experimental support for some FreeBSD capabilities. In the spirit of "rough consensus and working code", runj serves as a testbed for future proposals to extend the specification. For now, the extensions are documented here.

You can use runj demo spec to generate an example config file for your bundle.

Once you have a config file, edit the root path and process args to your desired values.

Lifecycle

Create a container with runj create $ID $BUNDLE where $ID is the identifier you picked for your container and $BUNDLE is the bundle directory with a valid config.json.

Start your container with runj start $ID. The process defined in the config.json will be started.

Inspect the state of your container with runj state $ID.

Send a signal to your container process (or all processes in the container) with runj kill $ID.

Remove your container with runj delete $ID.

containerd

Along with the main runj OCI runtime, this repository also contains an experimental shim that can be used with containerd. The shim is available as containerd-shim-runj-v1 and can be used from the ctr command-line tool by specifying --runtime wtf.sbk.runj.v1.

containerd 1.5 or later is required as earlier versions do not have all the necessary patches for FreeBSD support. Additional functionality may be available in a development build of containerd. If you prefer to build from source, you can find the latest commits in the main branch of containerd.

OCI Image

A base OCI image for FreeBSD 12.1-RELEASE on the amd64 architecture is available in the Amazon ECR public gallery. You can pull the image with the ctr tool like this:

$ sudo ctr image pull public.ecr.aws/samuelkarp/freebsd:12.1-RELEASE

If you prefer to build your own image, need an image for a different architecture, or want to try out a different version of FreeBSD, runj contains a utility that can convert a FreeBSD root filesystem into an OCI image. You can download, convert, and import an image as follows:

$ runj demo download --output rootfs.txz
Found arch:  amd64
Found version:  12.1-RELEASE
Downloading image for amd64 12.1-RELEASE into rootfs.txz
[...output elided...]
$ runj demo oci-image --input rootfs.txz
Creating OCI image in file image.tar
extracting...
compressing...
computing layer digest...
writing blob sha256:f585dd296aa9697b5acaf9db7b40701a6377a3ccf4d29065cbfd3d2b80395733
writing blob sha256:413cc9413157f822242a4bb2c86ea50d20b8343964b5cf1d86182e132b51f78b
tar...
$ sudo ctr image import --index-name freebsd image.tar
unpacking freebsd (sha256:5ac2e259d1e84a9b955f7630ef499c8b6896f8409b6ac9d9a21542cb883387c0)...done

Running containers with ctr

With containerd, runj, and the containerd-shim-runj-v1 binary installed, you can use the ctr command-line tool to run containers like this:

$ sudo ctr run \
    --runtime wtf.sbk.runj.v1 \
    --rm \
    public.ecr.aws/samuelkarp/freebsd:13.1-RELEASE \
    my-container \
    sh -c 'echo "Hello from the container!"'
Hello from the container!

ctr can also be used to test the experimental FreeBSD-specific extensions by creating a runj.ext.json file as documented in oci.md and passing the path with --runtime-config-path. For example, to run a container interactively with access to the host's IPv4 networking stack (similar to the --net-host networking mode on Linux):

$ cat <<EOF >runj.ext.json
{"network":{"ipv4":{"mode":"inherit"}}}
EOF
$ sudo ctr run \
    --runtime wtf.sbk.runj.v1 \
    --rm \
    --tty \
    --runtime-config-path $(pwd)/runj.ext.json \
    public.ecr.aws/samuelkarp/freebsd:13.1-RELEASE \
    my-container \
    sh

Note that containerd and runj will not automatically create an /etc/resolv.conf file inside your container. If your container image does not include one, you may need to add one yourself for name resolution to function properly. A very simple /etc/resolv.conf file using Google's public DNS resolver is as follows:

nameserver 8.8.8.8

Implementation details

runj uses both FreeBSD's userland utilities for managing jails and jail-related syscalls. You must have working versions of jail(8), jls(8), jexec(8), and ps(1) installed on your system. runj kill makes use of the kill(1) command inside the jail's rootfs; if this command does not exist (or is not functional), runj kill will not work.

Building

runj builds largely with standard go build invocations, except for the integ-inside integration test helper which must be statically linked. A Makefile is available for use which correctly sets the expected build options.

The following targets are available:

  • all (or just make without additional arguments) - Build all binaries and generate a NOTICE file.
  • NOTICE - Generate the NOTICE file based on included Go dependencies.
  • install - Install the runj binaries to the standard filesystem locations.
  • lint - Run golangci-lint which includes a number of linters.
  • test - Run all unit tests.
  • integ-test - Run integration tests. Note that this target must be run as root as it creates jails, creates and configures network interfaces, and manipulates pf rules. It also expects working Internet access to reach 8.8.8.8 for verification of a working network.
  • clean - Remove built artifacts.

runj normally expects to be built from a git checkout so that appropriate revision and module information is built in. If building runj from an extracted tar instead, you may populate the REV_OVERRIDE file with an appropriate value as a substitute for the normal revision provided from git.

Contributing

Please see the contribution policy.

Future

Resource limits on FreeBSD can be configured using the kernel's RCTL interface. runj does not currently use this, but may add support for it via rctl(8) in the future.

License

runj itself is licensed under the same license as the FreeBSD project. Some dependencies are licensed under other terms. The OCI runtime specification and reference code is licensed under the Apache License, 2.0; copies of that reference code incorporated and modified in this repository remain under the original license.

runj's People

Contributors

akhramov avatar dch avatar dependabot[bot] avatar dfr avatar emaste avatar gizahnl avatar hdbhm avatar kit-ty-kate avatar samuelkarp avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

runj's Issues

TestHello leaks runj-entrypoint processes

$ ps aux | grep runj-entrypoint
sam   75911   0.0  0.1   12796  2312  0  S+   09:50        0:00.00 grep runj-entrypoint
root  10036   0.0  0.0  728272   516  1- I    10Dec23      0:00.79 runj-entrypoint integ-test-hooks /var/lib/runj/jails/integ-test-hooks/exec.fifo /integ-inside -test.
root   4251   0.0  0.0  728016   456  3- I     8Dec23      0:00.84 runj-entrypoint integ-test-hooks /var/lib/runj/jails/integ-test-hooks/exec.fifo /integ-inside -test.
root   4671   0.0  0.0  728272   460  4- I     8Dec23      0:00.84 runj-entrypoint integ-test-hooks /var/lib/runj/jails/integ-test-hooks/exec.fifo /integ-inside -test.
root   4984   0.0  0.0  728272   460  5- I     8Dec23      0:00.86 runj-entrypoint integ-test-hooks /var/lib/runj/jails/integ-test-hooks/exec.fifo /integ-inside -test.
root  33454   0.0  0.0  728272   460  6- I    20Dec23      0:00.41 runj-entrypoint integ-test-hooks /var/lib/runj/jails/integ-test-hooks/exec.fifo /integ-inside -test.
root  55763   0.0  0.1  728272  4308  7- I    21Dec23      0:00.38 runj-entrypoint integ-test-hooks /var/lib/runj/jails/integ-test-hooks/exec.fifo /integ-inside -test.

Add racct/rctl support

The OCI runtime spec has support for rlimits under the process configuration. However, these limits are fairly minimal. FreeBSD has more resource limits, which can be applied to jails through rctl. It might be useful to add support for these racct limits to runj, even though it is not in the OCI specification.

To accomplish this, we'd add a new configuration option, structured similarly to the existing "rlimits" option in the spec. Then, it is a matter of adding a function that formats the rctl rule and calls rctl from the operating system, which would be called in the create command. I have an example of what this would look like here: https://github.com/cyrilzhangfreebsd/runj/tree/racct

Alternatively, we could add a new configuration file separate from config.json to contain configuration that is outside of the OCI spec.

Finally, I understand if you consider this enhancement to be outside of the project's scope. If you would not mind, I'd appreciate if you could provide some feedback on the changes I linked above, as I would continue to make experimental out-of-scope changes on my fork.

Containers which have console specified don't actually get a console set as controlling

FreeBSD requires a process to be session leader of it's process group and requires the TIOCSTTY ioctl to be called on the console FD to actually set the console as controlling.
Before:

root@mergusserator:~/go/src/go.sbk.wtf/runj # ctr run --runtime wtf.sbk.runj.v1 --tty --rm public.ecr.aws/samuelkarp/freebsd:12.1-RELEASE testjail /bin/csh
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
# 

After:

root@mergusserator:/var/lib # ctr run --runtime wtf.sbk.runj.v1 --tty --rm public.ecr.aws/samuelkarp/freebsd:12.1-RELEASE testjail /bin/csh
#

runj demo invalid character in URL

When trying the demo command as specified in the README.md I get the following error:

Error: parse "http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.0-RELEASE\n/base.txz": net/url: invalid control character in URL

root@mergusserator:~/runj # ./bin/runj demo download --output rootfs.txz
Found arch:  amd64
Found version:  13.0-RELEASE

Downloading image for amd64 13.0-RELEASE
 into rootfs.txz
Error: parse "http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.0-RELEASE\n/base.txz": net/url: invalid control character in URL
Usage:
  runj demo download [flags]

Flags:
  -a, --architecture string   CPU architecture, like amd64
  -h, --help                  help for download
  -o, --output string         Output filename (default "rootfs.txz")
  -v, --version string        FreeBSD version, like 12-RELEASE

Containerd shim: sporadic rootfs mount failures

containerd shim sporadically fails to mount the rootfs [1, 2].

The error originates within the containerd mounter during the shell command execution:

mount_nullfs: /run/containerd/io.containerd.runtime.v2.task/buildkit/nbxh1elkrwu64xuir6pmmjrtw/rootfs: Resource deadlock avoided

[question] Linux container support?

Does/would runj support running linux containers? (aka: docker images?)

Obviously provided that the software in these containers does not make any syscalls not supported by the linuxulator.

Seeing that a lot of projects are (often unfortunately) moving to distributing Docker images and not even bothering to check platform compatibility with other OS's it would be a big boon if possible :)

That and VNET support would be amazing!

terminal discards first keypress

When running a container with a terminal (through ctr run --tty), the first keypress after the container starts running appears to be discarded.

runj demo oci-image generates an image that buildah can't parse

I'm not sure if this is a buildah issue or a runj issue but i'll start with here:

When generating an image with:

$ runj demo oci-image -a arm64 -v 13.1-RELEASE

Buildah fails after some time processing with:

$ buildah from oci-archive:/tmp/image.tar 
open /var/tmp/oci2475433674/blobs/sha256/3d694e6482e047cd61a4ddbd153307cef6cdf3d4351d84729a3a35d6699db5ed: no such file or directory

The image.tar generated contains the following files:

.
โ”œโ”€โ”€ blobs
โ”‚ย ย  โ”œโ”€โ”€ 27e61852c31bd49c2364d377534932749138c265d394857c5201180217f49ab5
โ”‚ย ย  โ”œโ”€โ”€ 3d694e6482e047cd61a4ddbd153307cef6cdf3d4351d84729a3a35d6699db5ed
โ”‚ย ย  โ””โ”€โ”€ ccea2ddb5f7be88e12014dbdd424f0d3cce53f7b72f3c6864a1875ce236986b8
โ”œโ”€โ”€ index.json
โ””โ”€โ”€ oci-layout

index.json (reformatted for clarity) contains:

{ "schemaVersion": 2,
  "manifests": [
    { "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:3d694e6482e047cd61a4ddbd153307cef6cdf3d4351d84729a3a35d6699db5ed",
      "size": 350,
      "platform":
        { "architecture": "arm64",
          "os": "freebsd"
        }
    }
  ]
}
$ file blobs/*
blobs/27e61852c31bd49c2364d377534932749138c265d394857c5201180217f49ab5: gzip compressed data, original size modulo 2^32 958320640
blobs/3d694e6482e047cd61a4ddbd153307cef6cdf3d4351d84729a3a35d6699db5ed: JSON data
blobs/ccea2ddb5f7be88e12014dbdd424f0d3cce53f7b72f3c6864a1875ce236986b8: JSON data

blobs/3d694e6482e047cd61a4ddbd153307cef6cdf3d4351d84729a3a35d6699db5ed contains:

{ "schemaVersion": 2,
  "config":
    { "mediaType": "application/vnd.oci.image.config.v1+json",
      "digest": "sha256:ccea2ddb5f7be88e12014dbdd424f0d3cce53f7b72f3c6864a1875ce236986b8",
      "size": 206
    },
    "layers": [
      { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
        "digest": "sha256:27e61852c31bd49c2364d377534932749138c265d394857c5201180217f49ab5",
        "size": 280935454
      }
    ]
}

blobs/ccea2ddb5f7be88e12014dbdd424f0d3cce53f7b72f3c6864a1875ce236986b8 contains:

{ "author": "runj \[email protected]\u003e",
  "architecture": "arm64",
  "os": "freebsd",
  "config": {},
  "rootfs":
    { "type": "layers",
      "diff_ids": ["sha256:7206d787de4f70d6349127a4045078a06577529abc2186ae62d6e42114453575"]
    }
}

cc @dfr just in case

Is only FreeBSD on scope?

There are other FreeBSD-based OSes like DragonFly BSD that will likely take a, somewhat, different approach wrt jails and that don't have a linux compatibility layer.

I'm wondering if such would be considered, ofc by submitting PRs ourselves.

Docker run lines don't report completion

I'm not sure if this is a problem with runj, the containerd shim, or with the moby port (from https://github.com/gizahNL/moby), but If I write this minimal Dockerfile:

FROM kwiat/freebsd:13.0-RELEASE
RUN echo Hello

Running docker build prints hello but doesn't detect that the command has completed:

$ docker build .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM kwiat/freebsd:13.0-RELEASE
 ---> e122726568cf
Step 2/2 : RUN echo Hello
 ---> Running in 1e48d64d35ef
Hello

It hangs at this point and does not advance. docker --debug produces this output:

DEBU[2021-11-04T13:00:28.328627186Z] Calling POST /v1.39/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&target=&ulimits=null&version=1
DEBU[2021-11-04T13:00:28.413436000Z] [BUILDER] Cache miss: [/bin/sh -c echo Hello]
DEBU[2021-11-04T13:00:28.413603500Z] [BUILDER] Command to be executed: [/bin/sh -c echo Hello]
DEBU[2021-11-04T13:00:28.414392803Z] [zfs] ID:274bd9ef-1340-47bb-adc7-066976dcd929 START zfs snapshot zroot/docker/ad672c56da999d925214772f741033aa664bd2c8cec20e94ba6bae3eb38c2758@414348503  storage-driver=zfs
DEBU[2021-11-04T13:00:28.502713430Z] [zfs] ID:274bd9ef-1340-47bb-adc7-066976dcd929 FINISH  storage-driver=zfs
DEBU[2021-11-04T13:00:28.502994831Z] [zfs] ID:a8bdc8fd-9d96-402d-b42c-94c8c66976e4 START zfs list -Hp -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/docker/ad672c56da999d925214772f741033aa664bd2c8cec20e94ba6bae3eb38c2758@414348503  storage-driver=zfs
DEBU[2021-11-04T13:00:28.512969568Z] [zfs] ID:a8bdc8fd-9d96-402d-b42c-94c8c66976e4 FINISH  storage-driver=zfs
DEBU[2021-11-04T13:00:28.513273669Z] [zfs] ID:12ff4766-914b-4c3c-8375-50e6919e7c0a START zfs clone -p -o mountpoint=legacy zroot/docker/ad672c56da999d925214772f741033aa664bd2c8cec20e94ba6bae3eb38c2758@414348503 zroot/docker/39062f794ae04547c1705126d5954e7b9577ec6101b2ff74360cbf195ee96920-init  storage-driver=zfs
DEBU[2021-11-04T13:00:28.664143327Z] [zfs] ID:12ff4766-914b-4c3c-8375-50e6919e7c0a FINISH  storage-driver=zfs
DEBU[2021-11-04T13:00:28.664453728Z] [zfs] ID:9e08b49b-6521-4236-aaf3-b53a5d6b3544 START zfs list -Hp -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/docker/39062f794ae04547c1705126d5954e7b9577ec6101b2ff74360cbf195ee96920-init  storage-driver=zfs
DEBU[2021-11-04T13:00:28.675227768Z] [zfs] ID:9e08b49b-6521-4236-aaf3-b53a5d6b3544 FINISH  storage-driver=zfs
DEBU[2021-11-04T13:00:28.675527769Z] [zfs] ID:3f560d50-ea3a-4b8c-bcb6-8be6822258e1 START zfs destroy -d zroot/docker/ad672c56da999d925214772f741033aa664bd2c8cec20e94ba6bae3eb38c2758@414348503  storage-driver=zfs
DEBU[2021-11-04T13:00:28.750228346Z] [zfs] ID:3f560d50-ea3a-4b8c-bcb6-8be6822258e1 FINISH  storage-driver=zfs
DEBU[2021-11-04T13:00:28.750539147Z] mount("zroot/docker/39062f794ae04547c1705126d5954e7b9577ec6101b2ff74360cbf195ee96920-init", "/var/lib/docker/zfs/graph/39062f794ae0-init", "")  storage-driver=zfs
DEBU[2021-11-04T13:00:28.769797418Z] unmount("/var/lib/docker/zfs/graph/39062f794ae0-init")  storage-driver=zfs
DEBU[2021-11-04T13:00:28.956046407Z] [zfs] ID:879c1ec5-3724-4f1d-aa87-3628df008abd START zfs snapshot zroot/docker/39062f794ae04547c1705126d5954e7b9577ec6101b2ff74360cbf195ee96920-init@955941707  storage-driver=zfs
DEBU[2021-11-04T13:00:29.055206674Z] [zfs] ID:879c1ec5-3724-4f1d-aa87-3628df008abd FINISH  storage-driver=zfs
DEBU[2021-11-04T13:00:29.055468475Z] [zfs] ID:ad1128ea-00b8-4646-82b4-0f18bf6558c6 START zfs list -Hp -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/docker/39062f794ae04547c1705126d5954e7b9577ec6101b2ff74360cbf195ee96920-init@955941707  storage-driver=zfs
DEBU[2021-11-04T13:00:29.065764313Z] [zfs] ID:ad1128ea-00b8-4646-82b4-0f18bf6558c6 FINISH  storage-driver=zfs
DEBU[2021-11-04T13:00:29.066018914Z] [zfs] ID:f1c7c9b4-3231-405c-a131-e7cfc3208efa START zfs clone -p -o mountpoint=legacy zroot/docker/39062f794ae04547c1705126d5954e7b9577ec6101b2ff74360cbf195ee96920-init@955941707 zroot/docker/39062f794ae04547c1705126d5954e7b9577ec6101b2ff74360cbf195ee96920  storage-driver=zfs
DEBU[2021-11-04T13:00:29.218950880Z] [zfs] ID:f1c7c9b4-3231-405c-a131-e7cfc3208efa FINISH  storage-driver=zfs
DEBU[2021-11-04T13:00:29.219252881Z] [zfs] ID:de01cf79-fc92-4818-b7c7-00ab6b024cbc START zfs list -Hp -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/docker/39062f794ae04547c1705126d5954e7b9577ec6101b2ff74360cbf195ee96920  storage-driver=zfs
DEBU[2021-11-04T13:00:29.229770120Z] [zfs] ID:de01cf79-fc92-4818-b7c7-00ab6b024cbc FINISH  storage-driver=zfs
DEBU[2021-11-04T13:00:29.230035021Z] [zfs] ID:611e4d62-1fa9-46b0-93df-7ba2581a0cea START zfs destroy -d zroot/docker/39062f794ae04547c1705126d5954e7b9577ec6101b2ff74360cbf195ee96920-init@955941707  storage-driver=zfs
DEBU[2021-11-04T13:00:29.297591771Z] [zfs] ID:611e4d62-1fa9-46b0-93df-7ba2581a0cea FINISH  storage-driver=zfs
DEBU[2021-11-04T13:00:29.332823401Z] mount("zroot/docker/39062f794ae04547c1705126d5954e7b9577ec6101b2ff74360cbf195ee96920", "/var/lib/docker/zfs/graph/39062f794ae0", "")  storage-driver=zfs
DEBU[2021-11-04T13:00:29.335554711Z] container mounted via layerStore: &{/var/lib/docker/zfs/graph/39062f794ae0 0x2f07380 0x2f07380}  container=ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8
DEBU[2021-11-04T13:00:29.335736912Z] unmount("/var/lib/docker/zfs/graph/39062f794ae0")  storage-driver=zfs
DEBU[2021-11-04T13:00:29.578775211Z] attach: stderr: begin
DEBU[2021-11-04T13:00:29.578975512Z] attach: stdout: begin
DEBU[2021-11-04T13:00:29.579322813Z] mount("zroot/docker/39062f794ae04547c1705126d5954e7b9577ec6101b2ff74360cbf195ee96920", "/var/lib/docker/zfs/graph/39062f794ae0", "")  storage-driver=zfs
DEBU[2021-11-04T13:00:29.582091023Z] container mounted via layerStore: &{/var/lib/docker/zfs/graph/39062f794ae0 0x2f07380 0x2f07380}  container=ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8
DEBU[2021-11-04T13:00:29.583157427Z] Assigning addresses for endpoint great_meninsky's interface on network bridge
DEBU[2021-11-04T13:00:29.583373328Z] RequestAddress(LocalDefault/172.18.0.0/16, <nil>, map[])
DEBU[2021-11-04T13:00:29.583444928Z] Request address PoolID:172.18.0.0/16 App: ipam/default/data, ID: LocalDefault/172.18.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:2 Serial:false PrefAddress:<nil>
DEBU[2021-11-04T13:00:29.700087760Z] Assigning addresses for endpoint great_meninsky's interface on network bridge
DEBU[2021-11-04T13:00:30.332606300Z] Programming external connectivity on endpoint great_meninsky (20bae97f80d9218dcf7075ed0e24f87d2ff60d284648df45bffe9543ac94ef51)
DEBU[2021-11-04T13:00:30.332813901Z] EnableService ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 START
DEBU[2021-11-04T13:00:30.332835701Z] EnableService ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 DONE
DEBU[2021-11-04T13:00:30.347904457Z] bundle dir created                            bundle=/var/run/docker/containerd/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 module=libcontainerd namespace=moby root=/var/lib/docker/zfs/graph/39062f794ae0
DEBU[2021-11-04T13:00:30.536242553Z] event                                         module=libcontainerd namespace=moby topic=/tasks/create
DEBU[2021-11-04T13:00:30.572995190Z] event                                         module=libcontainerd namespace=moby topic=/tasks/start
DEBU[2021-11-04T13:00:30.600271190Z] event                                         module=libcontainerd namespace=moby topic=/tasks/exit

At the same time, containerd -l debug produces this:

DEBU[2021-11-04T13:00:30.417187913Z] event published                               ns=moby topic=/containers/create type=containerd.events.ContainerCreate
time="2021-11-04T13:00:30.469033805Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 pid=88053
time="2021-11-04T13:00:30.482128553Z" level=warning msg=CREATE req="&CreateTaskRequest{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,Bundle:/run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,Rootfs:[]*Mount{},Terminal:false,Stdin:,Stdout:/var/run/docker/containerd/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8/init-stdout,Stderr:/var/run/docker/containerd/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8/init-stderr,Checkpoint:,ParentCheckpoint:,Options:&types1.Any{TypeUrl:containerd.linux.runc.CreateOptions,Value:[],XXX_unrecognized:[],},XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1
time="2021-11-04T13:00:30.485106464Z" level=warning msg="Starting runj create" args="[create ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 /run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8]" id=ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 runtime=wtf.sbk.runj.v1
time="2021-11-04T13:00:30.514434273Z" level=warning msg="PROCESSING EXIT!" pid=88054 runtime=wtf.sbk.runj.v1 status=0
time="2021-11-04T13:00:30.533023542Z" level=warning msg="PROCESSING EXIT!" pid=88059 runtime=wtf.sbk.runj.v1 status=0
time="2021-11-04T13:00:30.533109242Z" level=warning msg="entrypoint waiting!" pid=88058 runtime=wtf.sbk.runj.v1 state="&{1.0.2-runj-dev ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 created 88058 /run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 map[]}"
DEBU[2021-11-04T13:00:30.534959549Z] event forwarded                               ns=moby topic=/tasks/create type=containerd.events.TaskCreate
time="2021-11-04T13:00:30.536146553Z" level=warning msg=START req="&StartRequest{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,ExecID:,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1
time="2021-11-04T13:00:30.550123105Z" level=warning msg="PROCESSING EXIT!" pid=88060 runtime=wtf.sbk.runj.v1 status=0
time="2021-11-04T13:00:30.550349906Z" level=warning msg=START runtime=wtf.sbk.runj.v1 state="&{1.0.2-runj-dev ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 created 88058 /run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 map[]}"
time="2021-11-04T13:00:30.570951582Z" level=warning msg="START runj" runtime=wtf.sbk.runj.v1 state="&{1.0.2-runj-dev ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 created 88058 /run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 map[]}"
time="2021-11-04T13:00:30.571051982Z" level=warning msg="PROCESSING EXIT!" pid=88061 runtime=wtf.sbk.runj.v1 status=0
DEBU[2021-11-04T13:00:30.571806085Z] event forwarded                               ns=moby topic=/tasks/start type=containerd.events.TaskStart
time="2021-11-04T13:00:30.572645788Z" level=warning msg=STATE req="&StateRequest{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,ExecID:,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1
time="2021-11-04T13:00:30.575246698Z" level=warning msg="PROCESSING EXIT!" pid=88058 runtime=wtf.sbk.runj.v1 status=0
time="2021-11-04T13:00:30.575310098Z" level=warning msg="INIT EXITED!" pid=88058 runtime=wtf.sbk.runj.v1
time="2021-11-04T13:00:30.598661785Z" level=warning msg="PROCESSING EXIT!" pid=88063 runtime=wtf.sbk.runj.v1 status=1
DEBU[2021-11-04T13:00:30.599392287Z] event forwarded                               ns=moby topic=/tasks/exit type=containerd.events.TaskExit
time="2021-11-04T13:00:30.603399402Z" level=warning msg="PROCESSING EXIT!" pid=88062 runtime=wtf.sbk.runj.v1 status=0
time="2021-11-04T13:00:30.603500002Z" level=warning msg=STATE resp="&StateResponse{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,Bundle:/run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,Pid:0,Status:STOPPED,Stdin:,Stdout:,Stderr:,Terminal:false,ExitStatus:0,ExitedAt:0001-01-01 00:00:00 +0000 UTC,ExecID:,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1 state="&{1.0.2-runj-dev ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 stopped 0 /run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 map[]}"
time="2021-11-04T13:00:30.628730396Z" level=warning msg=STATE req="&StateRequest{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,ExecID:,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1
time="2021-11-04T13:00:30.643039049Z" level=warning msg="PROCESSING EXIT!" pid=88068 runtime=wtf.sbk.runj.v1 status=0
time="2021-11-04T13:00:30.643189149Z" level=warning msg=STATE resp="&StateResponse{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,Bundle:/run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,Pid:0,Status:STOPPED,Stdin:,Stdout:,Stderr:,Terminal:false,ExitStatus:0,ExitedAt:0001-01-01 00:00:00 +0000 UTC,ExecID:,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1 state="&{1.0.2-runj-dev ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 stopped 0 /run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 map[]}"
time="2021-11-04T13:00:30.645278257Z" level=warning msg=STATE req="&StateRequest{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,ExecID:,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1
time="2021-11-04T13:00:30.659217708Z" level=warning msg="PROCESSING EXIT!" pid=88069 runtime=wtf.sbk.runj.v1 status=0
time="2021-11-04T13:00:30.659272909Z" level=warning msg=STATE resp="&StateResponse{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,Bundle:/run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,Pid:0,Status:STOPPED,Stdin:,Stdout:,Stderr:,Terminal:false,ExitStatus:0,ExitedAt:0001-01-01 00:00:00 +0000 UTC,ExecID:,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1 state="&{1.0.2-runj-dev ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 stopped 0 /run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 map[]}"
time="2021-11-04T13:00:30.667798840Z" level=warning msg=STATE req="&StateRequest{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,ExecID:,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1
time="2021-11-04T13:00:30.682164593Z" level=warning msg="PROCESSING EXIT!" pid=88070 runtime=wtf.sbk.runj.v1 status=0
time="2021-11-04T13:00:30.682265594Z" level=warning msg=STATE resp="&StateResponse{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,Bundle:/run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,Pid:0,Status:STOPPED,Stdin:,Stdout:,Stderr:,Terminal:false,ExitStatus:0,ExitedAt:0001-01-01 00:00:00 +0000 UTC,ExecID:,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1 state="&{1.0.2-runj-dev ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 stopped 0 /run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 map[]}"
time="2021-11-04T13:00:30.684506602Z" level=warning msg=STATE req="&StateRequest{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,ExecID:,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1
time="2021-11-04T13:00:30.698438154Z" level=warning msg="PROCESSING EXIT!" pid=88071 runtime=wtf.sbk.runj.v1 status=0
time="2021-11-04T13:00:30.698646154Z" level=warning msg=STATE resp="&StateResponse{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,Bundle:/run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,Pid:0,Status:STOPPED,Stdin:,Stdout:,Stderr:,Terminal:false,ExitStatus:0,ExitedAt:0001-01-01 00:00:00 +0000 UTC,ExecID:,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1 state="&{1.0.2-runj-dev ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 stopped 0 /run/containerd/io.containerd.runtime.v2.task/moby/ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 map[]}"
time="2021-11-04T13:00:30.699841059Z" level=warning msg=DELETE req="&DeleteRequest{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,ExecID:,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1
time="2021-11-04T13:00:30.713187108Z" level=warning msg="PROCESSING EXIT!" pid=88072 runtime=wtf.sbk.runj.v1 status=1
time="2021-11-04T13:00:30.737121997Z" level=warning msg="PROCESSING EXIT!" pid=88073 runtime=wtf.sbk.runj.v1 status=0

It looks as if it's tracking two processes in the jail. I'm not sure why the second exits first (maybe they exit at about the same time and are reported in different orders?). At the end of this, I would expect runj to report that the command finished.

If I restart dockerd, it notices that it has a container that appears to be running that it doesn't know about and tries to kill it:

DEBU[2021-11-04T13:05:06.936629561Z] shutting down container considered alive by containerd  container=ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8 paused=false restarting=false running=true
DEBU[2021-11-04T13:05:06.936674861Z] Sending kill signal 15 to container ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8
INFO[2021-11-04T13:05:16.988973523Z] Container failed to exit within 10 seconds of signal 15 - using the force  container=ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8
DEBU[2021-11-04T13:05:16.989196524Z] Sending kill signal 9 to container ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8

This triggers the following output from containerd:

time="2021-11-04T13:05:06.964064362Z" level=warning msg=KILL req="&KillRequest{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,ExecID:,Signal:15,All:false,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1
time="2021-11-04T13:05:06.977719212Z" level=warning msg="PROCESSING EXIT!" pid=88095 runtime=wtf.sbk.runj.v1 status=1
time="2021-11-04T13:05:17.009418898Z" level=warning msg=STATE req="&StateRequest{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,ExecID:,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1
time="2021-11-04T13:05:17.023122049Z" level=warning msg="PROCESSING EXIT!" pid=88097 runtime=wtf.sbk.runj.v1 status=1
ERRO[2021-11-04T13:05:17.023730451Z] get state for ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8  error="invalid character 'E' looking for beginning of value: unknown"
WARN[2021-11-04T13:05:17.023882952Z] unknown status                                status=0
time="2021-11-04T13:05:17.025965860Z" level=warning msg=KILL req="&KillRequest{ID:ac1d3703e32454d690b89bef70daafcb2d78d78e208dbf99091b91f0cf8f43b8,ExecID:,Signal:9,All:false,XXX_unrecognized:[],}" runtime=wtf.sbk.runj.v1
time="2021-11-04T13:05:17.039224209Z" level=warning msg="PROCESSING EXIT!" pid=88098 runtime=wtf.sbk.runj.v1 status=1

Again, it looks as if runj is generating a log message indicating that the process has exited, but isn't reporting the exit to containerd?

I can kill the jails with jail -r.

@kit-ty-kate, I took me minimal docker file from one that you'd filed in an issue on the moby port, so it's presumably worked for you in the past?

Cirrus CI uses outdated FreeBSD release

The current Cirrus CI configuration uses FreeBSD 13.0, which reached its end of life August 31, 2022. This causes pipeline failures, because the integration tests try to download an image that does not longer exist on the official download servers.

I would be willing to provide a PR, changing the image to freebsd-13.2. Apart from updating image_family in .cirrus.yml, is there anything I should be aware of that would have to be done as well?

missing module from vendor ?

I'm updating the sysutils/runj port to the 20231221 snapshot, but getting a build error. I don't know enough go to understand what's missing, and how to fix this - any suggestions?

# excerpt from full build log https://pkg.skunkwerks.at/poudriere/data/current_x64-default/2024-01-04_16h17m38s/logs/errors/runj-g20231221.log
...

===>  Building runj-entrypoint from ./cmd/runj-entrypoint
github.com/containerd/console
go.sbk.wtf/runj/cmd/runj-entrypoint
===>  Building containerd-shim-runj-v1 from ./cmd/containerd-shim-runj-v1
go: finding module for package google.golang.org/genproto/googleapis/rpc/status
vendor/github.com/containerd/ttrpc/types.go:22:2: cannot query module due to -mod=vendor
*** Error code 1

Normally we'd use go:modules to build these but this isn't possible until there's a tagged version I think of runj.

Here's my current patched port:

https://paste.sr.ht/~dch/495ccdbef07186a00fd686bdb18e273819010ab3

Renaming the project

Hello! And then you for an excellent project!

Is it possible to rename this project to freebsd docker or focker in short? ๐Ÿ™

Also, do you have a roadmap in case I'd like to dig in and help out?

Import of gojail

I've written gojail as a tool for both runj & the Docker port that I'm working on. As we've talked about integrating gojail with the runj repo seems like the best step forward to evolve both.

As of now gojail is implemented in pure go, using the standard FreeBSD syscalls and simple parsing of jail parameters. Implementing via cgo, using only the libc jail syscalls or additionally also using libjail seems trivial and can be done if the consensus goes in that direction.

TODO (missing/uninplemented in gojail):
-proper validation of jail parameters, either via a static configuration or dynamically via pulling in the syscall mibs
-implementation of the RunIn command, I'm still on the fence on how to properly do this. One direction I think might be worth exploring is simply returning an exec.Cmd wrapping the supplied command with /usr/sbin/jexec, if golang/go#46259 makes it we could then via go version build tags drop the jexec

PR will follow

Document what is missing

It would be great to have a to-do list that people who want to contribute can look at and see what still needs doing. I am not sure that I fully understand the separation of concerns between containerd and runj, so I don't know if things are missing from runj because containerd implements them or if they still need doing.

support ipv6

similarly to IPv4 support we should have IPv6 support too :)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.