After upgrade of k3os to v0.9.0, boot fails with kernel panic - attempted to kill init about picl-k3os-image-generator

[sgielen]

Aaand... Checking if $ENTER_ROOT is nonempty, then calling with -a enter-root instead of -a /sbin/init got k3os to boot. Will update this task tomorrow with a proper fix.

[sgielen]

For new images, I'll push a new commit momentarily. If you want to upgrade existing images, then make these changes before upgrading to k3os v0.9.0, by mounting the SD card to a compatible operating system. (If you've already upgraded to k3os v0.9.0 and found this issue, these changes should also fix your boot.)

For the orangepipc2 image:

• Mount the only partition on the SD card to a directory called 'root'
• In root/boot/env.txt, add to the end of the existing line init=/sbin/init.preinit (there must be a space before init=).

For the raspberrypi image:

• Mount the first partition on the SD card to a directory called 'boot' and the second to 'root'.
• In boot/cmdline.txt, add to the end of the existing line init=/sbin/init.preinit (there must be a space before init=).

Then, on both images:

• Remove the file at root/sbin/init. Then, create a new symlink: ln -s k3os root/sbin/init.
• Create a file at root/sbin/init.preinit with the following contents.
• chmod +x root/sbin/init.preinit.

#!/bin/sh

set -e

# sleep a second for devices to settle
sleep 1

# if an alternative init exists, exec it first. This is useful for future
# updates, as the file you're reading right now is inaccessible after k3os
# boots.
if [ -x "/k3os/system/init.preinit" ]; then
       exec /k3os/system/init.preinit
fi

modprobe squashfs || true
exec /sbin/init

• Then, cleanly unmount all partitions and try to boot. This should come up normally (even on earlier versions of k3os), after which you can perform k3os-upgrade-rootfs.

[sgielen]

This looks like k3os requires an argument to run, now, while earlier it did not.

The init script just runs /sbin/k3os -- it looks like this needs to be /sbin/k3os rc now. Will attempt a fix.

[sgielen]

I changed it to /sbin/k3os rc. Now, the boot fails with these messages:

1970/01/01 00:00:02 error making directory /run: mkdir /run: read-only file system
1970/01/01 00:00:02 error mounting tmpfs to /run: no such file or directory
1970/01/01 00:00:02 error making directory /var/cache: mkdir /var/cache: read-only file system
[more such errors]
1970/01/01 00:00:02 error mounting dev to /dev: device or resource busy
1970/01/01 00:00:02 Failed to run /usr/sbin/mdev -s: fork/exec /usr/sbin/mdev: no such file or directory
1970/01/01 00:00:02 Failed to run hwclock: fork/exec /sbin/hwclock: no such file or directory

[sgielen]

Trying with this as init:

#!/bin/sh
modprobe squashfs
mount -o remount,rw /
if [ ! -f "/usr/sbin/mdev" ]; then
  echo "#!/bin/sh" >/usr/sbin/mdev
  echo "exit 0" >>/usr/sbin/mdev
  chmod +x /usr/sbin/mdev
fi
if [ ! -f "/sbin/hwclock" ]; then
  echo "#!/bin/sh" >/sbin/hwclock
  echo "exit 0" >>/sbin/hwclock
  chmod +x /sbin/hwclock
fi
exec /sbin/k3os rc

Unfortunately, already the first mount fails because of "mount: can't read '/proc/mounts': No such file or directory". Adding a "mount -t proc proc /proc" before. Also, init says it can't create /usr/sbin/mdev because of a non-existent directory, so adding a mkdir -p, too.

#!/bin/sh
modprobe squashfs
mount -t proc proc /proc
mount -o remount,rw /
if [ ! -f "/usr/sbin/mdev" ]; then
  mkdir -p /usr/sbin
  echo "#!/bin/sh" >/usr/sbin/mdev
  echo "exit 0" >>/usr/sbin/mdev
  chmod +x /usr/sbin/mdev
fi
if [ ! -f "/sbin/hwclock" ]; then
  echo "#!/bin/sh" >/sbin/hwclock
  echo "exit 0" >>/sbin/hwclock
  chmod +x /sbin/hwclock
fi
exec /sbin/k3os rc

All errors are solved except for this one, apparently still making the boot fail:

1970/01/01 00:00:02 error mounting dev to /dev: device or resource busy
...
[   2.431823] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000

[sgielen]

Seems to be this code: https://github.com/rancher/k3os/blob/master/pkg/cli/rc/rc.go#L173

From there, it looks like the output are just warnings and not the cause of init being killed.

This was changed in rancher/k3os@a7a4688, but I'm not sure from that commit how to call /sbin/k3os as init now...

[sgielen]

Apparently, since that commit, /sbin/k3os cares whether it is being called with argv[0] set to init or /sbin/init. I should use exec -a /sbin/init /sbin/k3os to get k3os to boot in init mode. However, it looks like k3os actually calls back into /sbin/init, expecting to find itself, but instead finding our wrapper. When I add an echo at the start of /sbin/init, I get this kind of output:

Starting as PID 1.
[    1.849760] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
Exec k3os...
[   12.366798] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
Starting as PID 1.
Exec k3os...
[   22.836007] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
Starting as PID 1.

It looks like k3os (when run as init) starts by remounting root, then exec'ing /sbin/init again?

The source indeed seems to check whether it's called as /sbin/init, then runs the initrd function if so, which calls transferroot.Relocate() which transfers the root filesystem and reexecs itself with some flags in environment. This should be visible in busybox env.

Our init is then called again, calling k3os again with the same changed environment, so k3os should behave correctly and return from transferroot.Relocate() into the rest of the initrd function. Then, it remounts root and calls into enterchroot.Mount which then calls itself with argv[0] being enter-root. Unfortunately, $0 is always set to the script's filename, so I need to find a way to check whether I'm being re-execed as enter-root in order to exec -a enter-root /sbin/k3os.

[sgielen]

I wonder at this point, though, whether it wouldn't be easier to use init=/sbin/my-init instead of replacing /sbin/init. I might be able to do this for new images, but it might be difficult to do this for existing ones on both supported platforms.

[sgielen]

The first time our init runs its environment is:

SHLVL=1
HOME=/
TERM=linux
PATH=/sbin:/usr/sbin:/bin:/usr/bin
PWD=/

Subsequent times, the following variables are added/changed:

SHLVL=2
ENTER_DEVICE=/dev/loop0
ENTER_DATA=./k3os/data
ENTER_ROOT=/proc/self/exe

So I can use those to use exec -a enter-root instead of exec -a /sbin/init. It's a hack on top of a hack, but allright.

[sgielen]

So setting init to something like this works:

#!/bin/sh
modprobe squashfs
if [ -z "$ENTER_ROOT" ]; then
  exec -a /sbin/init /sbin/k3os
else
  exec -a enter-root /sbin/k3os
fi

But, I don't really like it and it will likely break in future versions of k3os. So, instead, I revisited my earlier comment about setting init= in the cmdline and realized that I already do this for init.resizefs. So, I'll go with reinstalling /sbin/init as a symlink to k3os, but set a cmdline to init=/sbin/init.preinit and make that a tool that calls /sbin/init, unless we override it at runtime to run something else:

#!/bin/sh

set -e

# sleep a second for devices to settle
sleep 1

# if an alternative init exists, exec it first. This is useful for future
# updates, as the file you're reading right now is inaccessible after k3os
# boots.
if [ -x "/k3os/system/init.preinit" ]; then
       exec /k3os/system/init.preinit
fi

modprobe squashfs || true
exec /sbin/init

With this, we keep a modprobe squashfs (which k3os actually performs itself as well according to the source code, but it doesn't seem to work for me?) and even give ourselves upgrade capability if we need it in the future.

[sgielen]

For new images, the issue is resolved by commit 5670894. For existing ones, see the previous coment.