Unable to boot newly created image about picl-k3os-image-generator

[sgielen]

Error -2 is ENOENT, indicating it can't find /sbin/init.resizefs (or its shebang, /bin/sh). This file only exists on first boot: it resizes the root volume, sets the proper init in cmdline, then deletes itself and its dependencies.

Do you have screen output when your Raspberry Pi boots? The first time it boots, the output should contain something like "Resizing root filesystem on..." - subsequent boots don't have this line in the output and immediately boot into k3os.

Could you write the image to your SD card again, and see if you can get the output of the first boot? If that works fine, but the second boot fails, something is going wrong with updating the cmdline. In that case, I'm interested in the contents of your /boot/cmdline.txt file.

If the first boot already gives the error you indicate, could you write the image freshly to the SD card and inspect of /sbin/init.resizefs does exist then?

[cwoolum]

Here's the screenshot of first boot.

It turns out /sbin/init.resizefs does exist. The tool I was using to explore the image was not working correctly.

I tried creating the image without docker as well. The udevadm warnings go away but I still get the error.

[cwoolum]

This is my init.resizefs

Click here to open

#!/bin/sh

set -e

IMAGE_TYPE="raspberrypi"

# sleep a second for devices to settle
sleep 1

mount -t proc none /proc
mount -t sysfs none /sys

# Unpack tools for resizing root FS
mount -t tmpfs -o size=50m none /tmp
tar -xJf /root-resize.tar.xz --strip 1 -C /tmp

export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/tmp/sbin:/tmp/bin:/tmp/usr/sbin:/tmp/usr/bin
export LD_LIBRARY_PATH=/tmp/lib:/tmp/lib/aarch64-linux-gnu:/tmp/usr/lib:/tmp/usr/lib/aarch64-linux-gnu

if [ "$IMAGE_TYPE" = "raspberrypi" ]; then
	PARTUUID=$(cat /proc/cmdline | sed -r 's/^.*PARTUUID=([^ ]+).*$/\1/')
	ROOTDEVICE=$(blkid | grep PARTUUID=\"$PARTUUID\" | awk -F: '{print $1}')
elif [ "$IMAGE_TYPE" = "orangepipc2" ]; then
	ROOTDEVICE=$(cat /proc/cmdline | sed -r 's/.*root=([^ ]+) .*/\1/')
fi
ROOTDISK=/dev/$(basename $(dirname $(readlink /sys/class/block/$(basename $ROOTDEVICE))))
PARTNUM=$(cat /sys/class/block/$(basename $ROOTDEVICE)/partition)

echo "== Performing filesystem check on root device $ROOTDEVICE... =="
e2fsck -pv $ROOTDEVICE
sleep 5
echo "== Resizing root filesystem on $ROOTDISK... =="
parted -s $ROOTDISK resizepart $PARTNUM 100%
sleep 5
mount -o remount,rw /
resize2fs $ROOTDEVICE
sleep 5

echo "== Setting proper init... =="
if [ "$IMAGE_TYPE" = "raspberrypi" ]; then
	mount ${ROOTDISK}p1 /boot
	sed -i 's# init=/sbin/init.resizefs# init=/sbin/init.preinit#' /boot/cmdline.txt
	NEW_PARTUUID=$(blkid -o export $ROOTDEVICE | grep PARTUUID)
	sed -i "s# root=PARTUUID=$PARTUUID # root=$NEW_PARTUUID #" /boot/cmdline.txt
	umount /boot
elif [ "$IMAGE_TYPE" = "orangepipc2" ]; then
	sed -i 's# init=/sbin/init.resizefs# init=/sbin/init.preinit#' /boot/env.txt
fi

echo "== Linking correct config... =="
if [ "$IMAGE_TYPE" = "orangepipc2" ]; then
	modprobe dwmac_sun8i
	sleep 1
fi

# TODO: what if this device doesn't have eth0, but has other interfaces? Loop through them?
MY_MAC=$(cat /sys/class/net/eth0/address)
if [ -r "/k3os/system/config/${MY_MAC}.yaml" ]; then
	ln -s /k3os/system/config/${MY_MAC}.yaml /k3os/system/config.yaml
else
	# TODO: retrieve config from a master?
	echo "Error: No configuration found for MAC ${MY_MAC}." >&2
	echo "Ensure that config/${MY_MAC}.yaml exists." >&2
	exit 1
fi

echo "== Cleaning up and rebooting... =="
rm /root-resize.tar.xz
rm /sbin/init.resizefs
# exec into another script so that /sbin/init.resizefs (this script) can be deleted,
# so the filesystem can be mounted read-only
cat <<EOF >/tmp/reboot
#!/bin/sh
set -e
sync
sleep 5
mount -o remount,ro / || echo u > /proc/sysrq-trigger || true
sleep 5
e2fsck -fyv $ROOTDEVICE || true
sleep 5
reboot -f
EOF
chmod +x /tmp/reboot
exec /tmp/reboot

[sgielen]

So, if I understand correctly, after you create an image, flash it to a USB drive and then immediately mount the partition (without booting the Raspberry Pi with it), you don't have a file at /sbin/init.resizefs?

Just to be sure: you are flashing the image picl-k3os-v0.10.0-raspberrypi.img and its last modified date is approximately when the Docker run process finished?

Could you check whether you have a file at /sbin/init.preinit on the flashed partition? Also, could you check if you see a /root-resize.tar.xz file?

These files are copied in build-image.sh around line 340:

sudo tar -cJf root/root-resize.tar.xz "root-resize"
sudo rm -rf root-resize

## Write a resizing init and a pre-init
sudo install -m 0755 -o root -g root init.preinit init.resizefs root/sbin                                                                                 
sudo sed -i "s#@IMAGE_TYPE@#$IMAGE_TYPE#" root/sbin/init.resizefs root/sbin/init.preinit

I wonder if you put a ls -la root/sbin/init* there, what the resulting output is.

[sgielen]

Oh, wait, I re-read your comment now and I understand that /sbin/init.resizefs does exist on the SD card, but your boot still fails with "Requested init /sbin/init.resizefs failed (error -2)."

That only makes slightly more sense, though, let me look into that.

[sgielen]

Could you modify your /boot/cmdline.txt to change init=/sbin/init.resizefs to init=/bin/sh? When you boot this, it should spawn a shell that allows you to interact. It'll probably just give you a "# " prompt.

If it fails as well, could you try removing the init=... from the line, then at the end of the line adding init=/bin/busybox -- sh? Contents of the file should then look something like dwc_otg.lpm_enable=0 [....] rootwait ro init=/bin/busybox -- sh. Then try booting that again. You don't need to reflash the image in between.

Once you get that, could you run ls -la /sbin/init.resizefs, ls -la /bin/sh and ls -la /bin/busybox to see what the state of those files are? Then, after that, could you run /sbin/init.resizefs to see what happens?

[sgielen]

Since you're the first person reporting you're building the image on Windows, I'm suspecting it has something to do with that; my first hypothesis is that it might have something to do with the execute bits on one of those three files, or with the fact that /bin/sh should be a symlink to busybox and that's being messed up somewhere.

If init=/bin/busybox doesn't work as intended, I'd also be interested in the state of the /sbin/init.resizefs, /bin/sh and /bin/busybox files after flashing the image. Bear in mind that /bin/busybox is a binary file, so its contents aren't very useful, and /bin/sh should be a symlink, so the type of file and the place it points to is more important than its contents.

If you want, you could also upload your generated .img file somewhere and I'll take a look at it.

[cwoolum]

Okay, I was able to get a bash prompt. When I run ls -la /sbin/init.resizefs, I get

-rwxr-xr-x    1 0            0                2708 Jul 29  2020 /sbin/init.resizefs

When I try to execute it though, I get not found

[cwoolum]

It looks like the line endings are wrong on init.resizefs which is probably why it's failing to execute. I ran file root/sbin/init.resizefs and got back

root/sbin/init.resizefs: POSIX shell script, ASCII text executable, with CRLF line terminators

I'm running this in WSL which uses an actual linux kernel. I know there are some low level discrepancies between that and actually running on linux but I didn't expect that the files would have windows line endings.

[cwoolum]

So I re-cloned this repo in WSL and re-ran the script and the line endings are correct now. It now is able to resize the volume.

I had previously cloned it in Windows and then ran it from Linux which is why the line endings were wrong.

[sgielen]

Interesting, very nice that you found the root cause! It makes sense that line endings caused this, and I wouldn't have thought of that myself. https://unix.stackexchange.com/questions/108588/shebang-line-not-working-with-cr-lf

I'd like to reopen this, because I think it should be easy to fix even when cloning natively on Windows. If I prepare a test branch, would you be willing to test if images built with that branch do work?

[sgielen]

@cwoolum I've pushed a possible fix to the fix/dos-line-endings branch. Would you be so kind as to check whether checking out the repo on native Windows, switching to the branch, building as you did before and then flashing & booting the image does work now?

[cwoolum]

So I did test your changes. While it does fix the line endings in the two init.* files, it does't fix the line endings on the *.sh scripts so I had to fix the line endings before I could even build the image.

I created a PR #23 that adds a .gitattributes. This forces windows to keep line endings for the files specified.

[cwoolum]

@sgielen, did you get a chance to look at my PR?

[sgielen]

Sorry for getting back late to this, @cwoolum.

PR #25 landed the cgroup changes, and since they were included in your PR #23 as well, it made it unmergeable. Therefore, I took your changes and committed them, giving credit to you in the log message (f1693d1). That should resolve your issue. If not, please reopen!