zbm-dev / zfsbootmenu

ZFS Bootloader for root-on-ZFS systems with support for snapshots and native full disk encryption

ZFSBootMenu is a Linux bootloader that attempts to provide an experience similar to FreeBSD's bootloader. By taking advantage of ZFS features, it allows a user to have multiple "boot environments" (with different distributions, for example), manipulate snapshots before booting, and, for the adventurous user, even bootstrap a system installation via zfs recv.

In essence, ZFSBootMenu is a small, self-contained Linux system that knows how to find other Linux kernels and initramfs images within ZFS filesystems. When a suitable kernel and initramfs are identified (either through an automatic process or direct user selection), ZFSBootMenu launches that kernel using the kexec command.

For more details, see:

• Documentation
• Boot Environments and You: A Primer

Join us on IRC

Come chat about ZFSBootMenu in #zfsbootmenu on libera.chat

zfsbootmenu's Issues

Failed to boot with "unary operator expected" error.

Line 25 of zfsbootmenu.sh has this:

if [ ${rootok} -eq 1 ]; then
  modprobe zfs 2>/dev/null
  udevadm settle
fi

Upon trying to boot the menu image I got it complaining about "-eq: unary operator expected" which implies that rootok was unset.

/usr/lib/udev is hardcoded

When I run generate-zbm on Gentoo (openrc, USE=split-usr in case that makes a difference), I get:

# generate-zbm
Mounting /boot/efi
Creating ZFSBootMenu 1.8.1 from kernel /boot/vmlinuz-5.4.51-gentoo-x86_64
dracut-install: ERROR: installing '/usr/lib/udev/vdev_id'
dracut: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.djkjGC/initramfs -a /usr/lib/udev/vdev_id
dracut: failed to install essential executable '/usr/lib/udev/vdev_id'
Failed to create /tmp/mo7y7dzWV3/zfsbootmenu.efi
Unmounting /boot/efi

I see that /usr/lib/udev is hardcoded in https://github.com/zbm-dev/zfsbootmenu/blob/ac8a8edafbbcd8e22013de676982edcd590445d9/90zfsbootmenu/module-setup.sh . If I change every /usr/lib/udev to /lib/udev, the problem goes away.

Build command line for other systems

Obarun (Arch) is unable to boot, because they use a slightly different zfs kernel command line.

root=zfs=obarun/ROOT/master

/etc/os-release

NAME="Obarun Linux"
PRETTY_NAME="Obarun Linux"
ID=obarun
ID_LIKE=arch
ANSI_COLOR="0;36"
HOME_URL="https://web.obarun.org/"
DOCUMENTATION_URL="https://wiki.obarun.org/"
SUPPORT_URL="https://forum.obarun.org/"
BUG_REPORT_URL="https://forum.obarun.org/"

Add a debug flag to show what commands are being run

This is not essential and I can try to implement when I get the time, would just be nice to have that, both for development as well as UX.

List of boot environments needs proper refresh after emergency shell

In the current implementation, a list of boot environments is written to ${BASE}/env before entering the menu input loop in zfsbootmenu.sh. Functions that are expected to alter this list, like clone_snapshot and (in the send-recv branch for now) duplicate_snapshot, modify the list by appending to ${BASE}/env when new boot environments are created.

If the user drops to an emergency shell with Alt+R and renames, creates or deletes BEs manually, ${BASE}/env will be out of sync with reality. If the generation of ${BASE}/env is moved to a function, complete with a drop to an emergency shell if no boot environments are found, there are two good possibilities for addressing this issue:

1. Call this function in every iteration of the menu input loop. This might slow things down a bit and is often unnecessary, but the list will always be accurate and there will be no need for clone_snapshot or duplicate_snapshot to update the list when they add a new boot environment.
2. Leave the {clone,duplicate}_snapshot functions to update the list as they do no, and only call the BE-list-update function after entering an emergency shell. This will be more efficient, but might not be the most proper approach for some definition of proper.

Relocate some scripts to /libexec

In pushing to support remote control of ZBM over SSH and dynamic startup and shutdown hooks, I put the zfsbootmenu-countdown script and the hook directories under /libexec because these scripts are not intended to be run by the user, and user interaction through the shell takes on greater significance over SSH. It would be nice to carry this over to the other non-interactive scripts:

• zfsbootmenu-input
• zfsbootmenu-preview.sh (why does this get a .sh extension when the others have them stripped?)

That leaves zfs-chroot (which I forget exists, but is nice to have), zfsbootmenu-help and zfsbootmenu in /bin where they belong.

Move colorize() to zfsbootmenu-lib.sh

Instead of embedding this in zfsbootmenu-help.sh, this function should be available for everything in the ZBM ecosystem to use - notably zfs-chroot.

Add option to rename ZFS filesystem

After cloning a snapshot, the name of the filesystem can get quite huge.

trident/ROOT/initial_pre-update-2020-02-01-18-44_000_pre-update-2020-02-03-08-37_000_pre-update-2020-02-03-14-48_001@pre-update-2020-02-03-20-14

An option should exist to easily rename a given filesystem to a new name.

Investigate snapshot cloning

Cloning a snapshot silently failed with 0.7.4. Determine if there's been a regression, or if the snapshot state on my boot pool is not being handled correctly.

Use dracut --uefi instead of objcopy

We talked about this on IRC, just leaving the issue here so I don't forget about it.

Pros:

• Drives objcopy for you
• Can sign the resultant image for Secure Boot if you define the keys in the config file

Cons:

• ?

Menu mapping doesnt respect custom keymap

I use https://github.com/eoli3n/void-config/blob/master/scripts/install/02-install.sh#L148 to configure azerty keymap for zfsbootmenu, which works, as keymap is azerty when i type my passphrase.

# Config keymap as https://github.com/zbm-dev/zfsbootmenu/issues/96#issuecomment-745627427
mkdir -p /mnt/etc/cmdline.d
echo 'rd.vconsole.keymap=fr' > /mnt/etc/cmdline.d/keymap.conf
echo 'install_optional_items+=" /etc/cmdline.d/keymap.conf "' > /mnt/etc/zfsbootmenu/dracut.conf.d/keymap.conf

Except, you told me to use an undocumented [ALT+W] combo to mount RW.
It didn't work, but [ALT+Z] did.
Menu alt combos doesnt respect keymap.

Add menu option to set new default BE

Add a shortcut option to the main menu to set the selected BE as the new default environment.

All snapshots feature is possibly not working

Users are reporting that when they hit alt-A on the main screen, an empty list is displayed when they know they have BE snapshots.

{FEATURE REQUEST} customized backgroud pic

Having the ability to custiomize the background pic, instead of the blank one now, would be nice

Handle kernel files without a version suffix

Arch and derivatives do not add a version suffix to their kernels in /boot. This breaks generate-zbm when attempting to set kver. Logic to handle this use case will have to be added - possibly by just not appending --kver to the dracut command line.

`kexec_kernel` only exports the pool containing the selected BE

We do a good job of cleaning up ZFS mounts throughout ZBM, mounting and unmounting only around the specific parts where we need filesystem access. We are not so careful with r/w pool imports (and cannot be, unless we want to annoy users with endless encryption prompts). Once a pool is imported r/w, it stays that way unless explicitly reimported.

In kexec_kernel, we export the pool containing the selected BE if that pool is mounted r/w. Best practice is probably to loop through all pools and export every r/w pool before doing the final kexec. This might require some care in case one or more exports fail or, if possible, the kexec were to fail.

Unable to unlock zpool with keylocation pointing to partition on USB stick

The use-case is to leverage a decryption key stored on a removable USB stick for a headless server by making use of this little trick. Admittedly this doesn't appear to be properly supported by ZFS at present, since it relies on setting keyformat=passphrase rather than smth more appropriate, such as raw. Nevertheless this works a treat: a pool set up in this manner unlocks automatically so long as the suitable USB stick is plugged into the box at boot time.

Unfortunately, ZBM struggles somewhat with this. As far as I can tell, the issue is that load-key() in zfsbootmenu-lib.sh wants to actually find the file referenced in keylocation; if it can't, it drops down to the penultimate line in that function, which presents a prompt.

I've implemented a workaround, which is to run zfs load-key -a at the beginning of load-key() (before the test that checks if the filesystem is actually encrypted). This works fine for my use case but I'm not sure if this could cause any unpleasant side-effects in other scenarios? Is there an issue with attempting to load all available keys (in the absence of any, perhaps?) and/or could this load too many keys?

I did try the suggestion to replace the -f at https://github.com/zbm-dev/zfsbootmenu/blob/master/90zfsbootmenu/zfsbootmenu-lib.sh#L1381 with a -e, to no avail.

Here's some info about the encrypted pool's setup:

# zfs list -o name,mountpoint,encroot,keystatus,keylocation,keyformat
NAME                                MOUNTPOINT                ENCROOT    KEYSTATUS  KEYLOCATION                           KEYFORMAT
apool                               none                      apool      available  file:///dev/disk/by-partlabel/my-key  passphrase
apool/HOME                          none                      apool      available  none                                  passphrase
apool/HOME/root_2021-01-02T11:20    /root                     apool      available  none                                  passphrase
apool/ROOT                          none                      apool      available  none                                  passphrase
apool/ROOT/ubuntu_2021-01-02T11:20  /                         apool      available  none                                  passphrase

Without the workaround, in ZBM's emergency shell after I beat the passphrase prompt about 5 times, zpool list shows the pool online (unlocked) and the datasets are also shown; however, unless I issue zfs load-key -s in the emergency shell, I can't boot.

More context can be found here.

[Feature request] Make the snapshot screen more dummy proof

• Have help texts explaining each option (WIP)
• Add job control to the duplicate process, otherwise a user can get stuck waiting for it to complete
• Check available space before duplicating a dataset
• Make it so that if a snapshot from the default bootfs is cloned and promoted, the new dataset should become the default bootfs; for added points, have an option to do automatic renaming so the promoted snapshot keeps the name.

How would one set a keyboard layout for `zfsbootmenu`?

I use the Dvorak layout—I know my password on the normal US layout, but it is still a bit of an impedance sometimes.

Add option to edit kernel command line

Before booting a system, we should present the option to edit the kernel command line. This will require the following:

1. As each boot environment is detected, find the kernel command line then and store it in a specific file
2. Add a shortcut to present a prompt with the file contents pre-filled, so that they can be edited
3. Save the input string back to the kernel command line file

How do you handle swap and/or hibernation with voidlinux?

The subject says it all. I'm on laptop and right now I'm using a zvol for the swap, but I can't hibernate with such a setting.

Do you use a second partition and put the swap on it? Would hibernation work then?

wiki: outdated instructions for config.yaml

Hi,

Thanks a lot for this nice boot menu and the detailed instructions in the wiki. I followed the "Single disk install" and noticed that the settings in /etc/zfsbootmenu/config.yaml have been changed and are not consistent anymore with the instructions.

cheers,
Pascal

First install tutorial

Hey, I am coming from FreeBSD and OpenBSD and I would like to install VoidLinux on my Laptop with ZFS-encrypted home and unencrypted rest.
My Linux knowledge is quite rusty but I had hoped to find a good SystemD-free setup that is ZFS friendly with VoidLinux and your bootmenu! But so far I can't get the system setup. The multitude of configuration options and different bootloaders etc. is quite daunting when one is used to *BSD.

I tried to follow https://github.com/nightah/void-install but things broke once I encrypted the /home -- I wanted to add zfsbootmenu after the install.
It seems like the whole partitioning has to happen differently, though...
Could you tell me which steps to adapt to get zfsbootmenu (and probably rEFInd?) setup directly? If it works, I could try to integrate it with above tutorial as a wiki page in this repository.

Thank you so much!

liunuxx64.efi.stub part of systemd on Arch

I'm aware that Arch is probably not going to work yet, but I'm trying a few things.

(I've manually made versioned kernel files to get this far)

The objcopy command fails because on Arch linuxx64.efi.stub is part of systemd now, and lives under /usr/lib/systemd/boot/efi/linuxx64.efi.stub

Add checks for objcopy and gummiboot

When making a unified EFI file, check if these pre-reqs are present on the system and gracefully bail if not.

Failed to start Switch Root

I installed Debian Buster with root on ZFS with rEFInd as UEFI boot manager, dracut as initramfs generator, systemd as init system. Everything was ok so i tried to install zfsbootmenu (i cloned repo, make installed, installed perl libs, kexec-tools, fzf, mbuffer) pasted config, set zfs property, generated boot environment (generate-zbm), and rebooted. Boot failed with Failed to start Switch Root error. So my question is there something i've done wrong? Boot without zfsbootmenu works.
My ZFS topology

NAME	MOUNTPOINT	CANMOUNT
zroot	none	on
zroot/ROOT	none	off
zroot/ROOT/debian.d2020.09.18	/	noauto
zroot/home	/home	on

zroot is encrypted, keylocation=prompt

NAME	PROPERTY	VALUE	SOURCE
zroot/ROOT	org.zfsbootmenu:commandline	spl_hostid=2c9a715c ro quiet	local

I've tried to copy full rdsosreport.txt but there were no sda1 in /dev to mount. I don't know any other way to save it.

End of journal

dracut state

refind_linux.conf

"Boot default" "zfsbootmenu:ROOT=zroot spl_hostid=2c9a715c timeout=0 ro"
"Boot to menu" "zfsbootmenu:ROOT=zroot spl_hostid=2c9a715c timeout=-1 ro"

zfsbootmenu config
config.txt

dracut conf
zol.txt

zfs-import.target zfs-mount.service zfs-import-cache zfs.target systemd units are enabled.

I can share Virtualbox VM if needed

Explore moving fzf/sk keybinds to the caller

Instead of hard-coding keybinds in the draw_ functions, possibly pass them in so they're defined closer to the caller.

Cloned filesystems do not have all properties set

A side-effect of cloning a snapshot into a new filesystem is that properties set on the snapshot are not transferred to the clone.

$ zfs get -H -o value org.zfsbootmenu:test zroot/ROOT/void.2020.06.16
test test test

$  zfs get -H -o value org.zfsbootmenu:test zroot/ROOT/void.2020.06.16@proptest
test test test

$ zfs clone zroot/ROOT/void.2020.06.16@proptest zroot/ROOT/void.2020.06.17

$ zfs get -H -o value org.zfsbootmenu:test zroot/ROOT/void.2020.06.17
-

There's some logic in beadm that can in large part be duplicated in zfsbootmenu, but it will add a substantial amount of extra code to be tested.

<dataset> "is already mounted" printing in a loop; the boot menu never loads

Hey, first off: Thanks for making this project! It's nice to know that there's other people that obsess over this stuff as much as I do.

I've been spending the last few days off and on debugging an issue I'm running into in which the zfsbootmenu Repeatedly prints "filesystem is already mounted".

Any idea what could be causing it? It's keeping me from booting my system right now :/ Enclosed is a screenshot of the error.

Thanks for your help, and this totally awesome project.

CTRL+E (edit kernel cmdline) no longer works on master version

This is a regression compared to version 1.8.0. Thankfully it's easy to fix:

zfsbootmenu/90zfsbootmenu/zfsbootmenu-lib.sh

Line 199 in ab95346

expects="--expect=alt-k,alt-d,alt-s,alt-c,alt-r,alt-p,alt-w,alt-i"

is missing alt-e,

Add index to cloned snapshots

Append _${i} to each cloned snapshot of the same filesystem, so that it can be done multiple times.

How to set final kernel parameters?

Thank you all for the speedy answers on the keymap issue, but one last question and I will be satisfied.

If I am understanding how ZBM works, the boot parameters set for ZBM do not affect the final boot after entering in the passphrase for the encrypted ZFS root. So then, how would one passthrough kernel parameters for the final boot?

Fix kernel detection routine in generate-zbm

When installing via hrmpf, the livecd will boot 5.2.x, but the new chroot'd install will have something much newer. If the kernel version returned by uname -r doesn't find anything, attempt to find the most recent kernel in /boot and extract the kernel version from that.

generate-zbm should be aware of free space

Project Trident users are noting that their EFI partitions are filling up and generate-zbm is storing incomplete/broken ZFSBootMenu initramfs images, rendering their systems unbootable. We should be aware of space constraints on the target volume, and act accordingly when there is insufficient space to store both the kernel and the initramfs.

Add toggle to mount/unmount EFI partition

Add a configuration option to mount the EFI partition at the start of generate-zbm, and unmount it at the end. MountEFI=/boot/efi or some such - this will rely on the appropriate fstab line existing.

Cloned snapshots need attention

Users are cloning older snapshots and booting them as full BEs. The cloned filesystems need:

• A better naming scheme instead of fs_snapshotname
• Assurance that they are created correctly and subsequently able to be deleted correctly.

Support different command lines with Secure Boot

Currently, only the UEFI bundle supports Secure Boot. Since the kernel command line is embedded into the generated image, I would need multiple images generated by zfsbootmenu, one for each variation of the command line. This is interesting because the command line used to boot the can only be one of the pre-configured ones.

The other option is to support signing the kernel itself, and then selecting the boot entry with the normal rEFInd procedure. This is cool, because it has flexibility, but it's not ideal for the security minded, since it allows one to edit the kernel command line to do god knows what.

Therefore, I think it would make sense to enable both of the options, instead of choosing one of them. For the first one, we'd need to find some way of conditionally defining the kernel command line (I suggest setting global variables when running dracut, and then defining the cmdline based on that). For the second one, we could drive sbsign manually or wait for sbctl release + integration.

Unable to detrmine kernel version from /boot/vmlinuz.old (Ubuntu)

When deploying new kernel versions, Ubuntu typically provides softlinks vmlinuz and vmlinuz.old that point to the current and previous kernels, as in:

# ll /boot/vmlinuz*
lrwxrwxrwx 1 root root       24 Jan  6 15:34 /boot/vmlinuz -> vmlinuz-5.4.0-59-generic
-rw------- 1 root root 11682560 Dec  9 08:30 /boot/vmlinuz-5.4.0-58-generic
-rw------- 1 root root 11686656 Dec 10 12:40 /boot/vmlinuz-5.4.0-59-generic
lrwxrwxrwx 1 root root       24 Jan  6 15:34 /boot/vmlinuz.old -> vmlinuz-5.4.0-58-generic

This causes an issue when running generate-zbm without args, where it attempts to determine the latest kernel version. The error message is as per the title of this ticket.

The fix is simple enough; exclude soft-links by inserting the following at line 532 of generate_zbm;

next if (-l $_);

This works Ubuntu; not sure if there are cases where soft-links should not be ignored in other distros?

Weird issue with encrypted pool

Ok, just updated to zbm 1.3, should try it with 1.2 to try and pinpoint it.

Current environment: 2 boot environments, under zroot/ROOT. I booted into zbm and poked around, then tried Alt+D for "select bootfs". It dropped me into a password prompt for zroot, for which my password didn't work, and I couldn't quit it without resorting to REISUB (without the R, though), which rebooted my device completely. Logging in normally worked perfectly, then.

Failure in Unified UEFI mode isn't propagated

If the user doesn't install gummiboot, the objcopy command fails, but the error isn't propagated. This results in xbps reporting "zfsbootmenu: configured successfully", which is false.

EDIT: That might be an issue with objcopy itself, seems like you included error checking.

Problem: file system already mounted

Hi,
I installed zfsbootmenu on VoidLinux VM.
The machine boots well via GRUB bootloader.
I generate zfsbootmenu image via generate-zbm command (EFI).
I add a new entry in efibootmgr.
When i boot it from Efi stub, there are two problems:

1. I see debug printouts before zfsbootmenu appears even though loglevel is set to 0 in config.yaml.
2. After entered password i see : filesystem already mounted and it cannot go on.

POOL

zpool status
  pool: zroot
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          vda2      ONLINE       0     0     0

errors: No known data errors

DATA SET

zroot                       6,97G  31,3G      192K  none
zroot/DATA                  16,8M  31,3G      192K  none
zroot/DATA/home             16,6M  31,3G     16,6M  /home
zroot/ROOT                  2,70G  31,3G      192K  none
zroot/ROOT/void.2020.08.30  2,70G  31,3G     2,70G  /
zroot/VAR                    556K  31,3G      192K  none
zroot/VAR/log                364K  31,3G      364K  /var/log
zroot/swap                  4,25G  35,5G      108K  -

Property canmount

zroot                       canmount               off                    local
zroot/DATA                  canmount               on                     default
zroot/DATA/home             canmount               on                     default
zroot/ROOT                  canmount               on                     local
zroot/ROOT/void.2020.08.30  canmount               noauto                 local
zroot/VAR                   canmount               on                     default
zroot/VAR/log               canmount               on                     default

As you can see, zroot/ROOT/void.2020.08.30 is set to noauto.

How can i resolve ?

Kernel version selector should have "set default" abilities

In the kernel selection menu invoked by ALT+K for a specific boot environment, there should be an ALT+D hotkey that pins the default kernel version for that environment, much like ALT+D at the top level sets a default boot environment. This amounts to setting the pool writable and setting the org.zfsbootmenu:kernel property on the current boot environment.

Unified UEFI images do not seem to honor kernel cmdline as expected

When producing a unified UEFI image with generate-zbm, the contents of CommandLine in the [Kernel] section of /etc/zfsbootmenu/config.ini are passed as the value for the the --kernel-cmdline argument to dracut. When the unified image is produced, the command-line arguments can be verified with lsinitrd /path/to/zbm/image. Everything seems to be done as expected, except there is a blank argument listed first, e.g.,

Command line:

rd.vconsole.font=ter-124b
nomodeset
i915.modeset=0
ro
quiet
loglevel=0

There is an alternative method for setting the kernel cmdline, by setting the kernel_cmdline parameter in a dracut configuration file such as /etc/zfsbootmenu/dracut.conf.d/cmdline.conf. When dracut runs, it concatenates the values of the kernel_cmdline parameter with the --kernel-cmdline argument passed by generate-zbm. If kernel_cmdline is nonempty, then there is no blank argument in the lsinitrd output. For example, if /etc/zfsbootmenu/config.ini contains CommandLine=zfsbootmenu ro quiet loglevel=1 and /etc/zfsbootmenu/dracut.conf.d/cmdline.conf contains kernel_cmdline="nomodeset i915.modeset=0", then lsinitrd will show

nomodeset
i915.modeset=0
zfsbootmenu
ro
quiet
loglevel=1

Either way, one expects these arguments to be used when the unified UEFI image is booted. However, that does not seem to be the case. For one, even with quiet loglevel=0, I see kernel messages flashing on screen before the boot menu starts. For another, if I drop to an emergency shell in zfsbootmenu and look at cat /proc/cmdline, I see nothing. The output of dmesg shows Command line: \x01. Some flags, like rd.vconsole.font, are properly respected when zfsbootmenu tries to set console options, but other flags that control driver options are not. I have a suspicion that the cmdline flags for dracut are for overriding or augmenting the cmdline seen by later-stage dracut modules and the subsequent boot process, not for setting the early-boot arguments that control the booting of the initial zbm kernel. Maybe rEFInd is the only workable approach for setting the cmdline when UEFI implementations don't support passing arguments directly.

Because @ericonr knows more about dracut and unified UEFI images, I'll pull him into this problem.

Encrypting /boot (evil maid attack)?

This is one of the most complete guides I've found:
https://gist.github.com/huntrar/e42aee630bee3295b2c671d098c81268

Any idea how your's would be modified to protect against the evil maid attack? I'd like to use the boot loaders you've already laid out.
Thanks!

cat: /tmp/zfs.lWmn/obarun/ROOT/master/kernels: No such file or directory

Hi

I'm trying to make zfsbootmenu to work on Obarun (Arch) but I have some problems.
I can boot and get the Refind menu:

By pressing F2 I get this information:

Then I get the zfsbootmenu:

First observation when I press ALT+K, the zfsbootmenu is empty:

When I press [ENTER] in this menu, this is what is displayed at the bottom of the screen:

Finally at the main menu when I press enter, I get this error:

My configuration files are available as attached files.

config.ini.txt
zfsbootmenu.conf.txt
refind_linux.conf.txt
fstab.txt
tree-boot.txt

Thank you
Best Regards

[FEATURE REQUEST] Support for Ubuntu

Please add support for Ubuntu as well. Starting from 19.04, the installer is able to install Ubuntu with zfs-on-root. Assuming that this is the canonical way, the standard for Ubuntu should be:

If your system does NOT use Dracut, I'll be happy to add support. Please open an issue with the following information:

Dracut is not used, Ubuntu/Debian relies on initramfs-tools instead.

How to detect your OS based on a file in your imported OS pool (contents of /etc/os-release would be a good start).

cat /etc/os-release

NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu Focal Fossa (development branch)"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

How to format the kernel command line for ZFS on your system

Boot stanza from the grub menu, autogenerated by the 10_linux_zfs GRUB file:

setparams 'Ubuntu Focal Fossa (development branch)'
        
        recordfail
        load_video
        gfxmode ${linux_gfxmode}
        insmod gzio
        if [ "${grub_platform}" = xen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod zfs
        set root='hd0,gpt4'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs=uuid --set=root --hint-bios=hd0,gpt4 --hint-efi=hd0,gpt4 --hint-baremetal=ahci0,gpt4 afa2440c1b0c14f5
        else
          search --no-floppy --fs=uuid --set=root afa2440c1b0c14f5
        fi
        linux        /BOOT/ubuntu_utwd0l@/vmlinuz-5.3.0-18-generic root=ZFS=rpool/ROOT/ubuntu_utwd0l ro  quiet splash ${vt_handoff}
        initrd        /BOOT/ubuntu_utwd0l@/initrd.img-5.3.0-18-generic

Where you store your default kernel command line parameters.

/etc/default/grub

safeCopy should optionally preserve mtime/atime

The current implementation of safeCopy always tries to preserve the mtime/atime of files it copies. When in components mode, a kernel is copied in from /boot . Depending on how you've shuffled / rebuilt in the past, the copied in kernel can have an mtime older than the kernel associated with the previous zfsbootmenu instance. This results in the new version of zfsbootmenu being ordered behind an older version by rEFInd. We should not preserve mtime/atime when creating new files for zfsbootmenu, but only when rotating through backups.

support request: how to create a bootenv ?

if I run

zfs snapshot zroot/ROOT/void@now
zfs clone zroot/ROOT/void@now zroot/ROOT/void-new
zfs promote zroot/ROOT/void-new
reboot

I only see in zfsbootmenu void dataset.
If i go to snapshot submenu, then clone and promote to a new name void-new2, it appears in list.

Also, how to chroot a dataset in rw mode from zfsbootmenu ?

Failing to boot (black screen) with rootfs-block omitted from dracut_modules

Commit cd8f889 is causing zfsbootmenu to black screen on my machine. Reverting this change fixes the issue on release 1.6.

Machine is a single-disk, single-zpool XPS15 laptop. Void Linux host. Dataset layout is a single zpool encompassing / (including /boot) on top of LUKS2 encryption.

Zfsbootmenu's dracut.conf.d is default except for a single addition:

omit_drivers+=" nouveau "

Command line passed to zfsbootmenu's kernel is:

zfsbootmenu:ROOT=korra-sys ro spl_hostid=949f4796 loglevel=7 zfs_force=1 timeout=5 nomodeset nouveau.modeset=0 rd.luks.uuid=7f25b7a9-cae2-463a-aa0e-2208bbe6a758

zfsbootmenu is packaged up into a unified and signed EFI boot file via gummiboot's efistub like this: http://sprunge.us/BOyk4c. I don't believe this is related to the issue.

Please let me know if there is any other information I can provide from my setup.

[FEATURE REQUEST] Forward command-line arguments to "client" kernels

Currently, zfsbootmenu looks in /etc/default/grub to find the command-line arguments it passes to "client" kernels it will boot. It might be more convenient and more transparent to forward the arguments passed to the zfsbootmenu kernel. This allows users to specify arguments in the "natural" place: the boot manager that would boot their kernels if zfsbootmenu were not used.

There are a couple of considerations here:

1. Some arguments will need to be culled or replaced when passing to clients (e.g., zfsbootmenu or root arguments);
2. If zfsbootmenu is using a different kernel version than the one selected for booting, the arguments may not be directly compatible.

The first point may not be that big a deal, because zfsbootmenu already takes responsibility for the root argument and removing the specific zfsbootmenu argument would be trivial. The second point may be more problematic, since a key feature of boot environments is the booting of different kernel versions. This admits the possibility that the zfsbootmenu and client kernels will have different versions and potentially different sets of supported arguments.