On an up-to-date TrueOS system with
% beadm --version
beadm 1.2.6 2015/09/17
I got this error:
% /usr/local/sbin/beadm list
awk: cmd. line:18: warning: escape sequence \/' treated as plain /'
awk: cmd. line:51: fatal: attempt to use scalar `FSNAMES' as an array
If I reset the PATH and retry, I get expected output:
% env PATH=/bin:/usr/bin:/usr/local/bin /usr/local/sbin/beadm list
BE Active Mountpoint Space Created
initial - - 7.7G 2016-09-08 04:47
12.0-CURRENT-up-20160912_225329 - - 4.3G 2016-09-12 22:48
12.0-CURRENT-up-20161226_222208 - - 149.3M 2016-12-26 22:21
12.0-CURRENT-up-20170112_231255 NR / 406.0G 2017-01-12 22:19
Which awk did the first instance use?

    % which awk
    /usr/uumath/bin/awk

    % /usr/uumath/bin/awk --version
    GNU Awk 4.1.4, API: 1.1 (GNU MPFR 3.1.4-p3, GNU MP 5.1.3)
    ...

The issue is in /usr/local/sbin/beadm: it has

    PATH=${PATH}:/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin

That is dangerous for installation at remote sites: it should instead
ignore the user's PATH variable, and supply a fixed one:

    PATH=/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin
    export PATH

That way, it will get the needed TrueOS/FreeBSD version of awk, rather
than a locally-installed GNU gawk that might have also been named awk.

The export command shown above should be there too: it is missing in
the current beadm.

I haven't inspected the awk code in beadm: it can probably be tweaked
slightly to work with gawk, mawk, and nawk, without needing to do
feature-tests to figure out which one is being used. I've written

0.5 million lines of awk, and with rare exceptions, almost all of my
awk code works on all three implementations.

Short version

Use canmount=noauto for all (even active) BEs; the boot loader uses bootfs and mount it anyway and off we go. This makes backups less error-prone.

Long version

The FreeBSD boot loader (tested on 10.3) will happily boot into and mount a canmount=noauto filesystem pointed to by the zpool's bootfs property. (By the by, it will happily mount a canmount=off filesystem pointed to by bootfs, too.) By setting all BEs to noauto -- rather than having the active one set to on in addition to being pointed to by bootfs -- a ZFS send/recv of the root pool or filesystems (BEs) to another location won't have the possibility of mounting over the root partition. (See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=210222)

Is there any drawback to setting canmount=noauto here? The only one I can think of is if this pool is imported from another (e.g. livecd) system with altroot; the resulting fs won't come up quite how you would "hope"; but I would suggest this is a recovery situation and is OK; performing a zfs send/recv to perform a backup of your system and being rewarded with havoc on reboot is much more disruptive.

This won't address any other mount points (/usr/home, or /var/log, etc.) that may be getting backed up from also needing special care, but those filesystems are much less likely to be appear after initial backups (unlike new BEs; especially if created with -e), and also are less disruptive (you have a good chance of umount-ing them) than stacking onto /.

Hi

From the usage example at https://vermaden.wordpress.com/2022/03/14/zfs-boot-environments-revolutions/, I assumed that reroot without prior activation would temporarily reroot – like, T but not R.

Actual result

Condensed:

root@mowa219-gjp4-8570p-freebsd:/usr/home/grahamperrin/dev/beadm # ./beadm create n253776-d5ad1713cc3-b
Created successfully
root@mowa219-gjp4-8570p-freebsd:/usr/home/grahamperrin/dev/beadm # script
Script started, output file is typescript
root@mowa219-gjp4-8570p-freebsd:/usr/home/grahamperrin/dev/beadm # ./beadm reroot n253776-d5ad1713cc3-b
Activated successfully
root@mowa219-gjp4-8570p-freebsd:/usr/home/grahamperrin/dev/beadm # 
…
% date ; uptime ; bectl list -c creation
Fri 18 Mar 2022 09:40:01 GMT
 9:40a.m.  up 3 days,  4:31, 2 users, load averages: 3.19, 1.73, 1.82
BE                    Active Mountpoint Space Created
n250511-5f73b3338ee-d -      -          4.94G 2021-11-13 15:43
n252381-75d20a5e386-b -      -          6.80G 2022-01-12 23:23
n252450-5efa7281a79-a -      -          6.49G 2022-01-14 19:27
n252483-c8f8299a230-b -      -          4.84G 2022-01-17 14:24
n252505-cc68614da82-a -      -          4.90G 2022-01-18 14:26
n252531-0ce7909cd0b-h -      -          5.71G 2022-02-06 12:24
n252997-b6724f7004c-c -      -          6.17G 2022-02-11 23:07
n253116-39a36707bd3-e -      -          5.66G 2022-02-20 07:03
n253343-9835900cb95-c -      -          1.54G 2022-02-27 14:58
n253627-25375b1415f-e -      -          462M  2022-03-12 18:20
n253776-d5ad1713cc3-a -      -          22.9M 2022-03-14 23:40
n253776-d5ad1713cc3-b NR     /          156G  2022-03-18 09:31
% 

The reroot alone resulted in R:

n253776-d5ad1713cc3-b NR / 156G 2022-03-18 09:31

As a default, R is not a bad thing :-) however there may be cases where T will be preferable.

Side notes:

• I wondered whether script(1) above would capture anything, of course it did not (because I did not explicitly exit before the reroot)
• the reroot feature is very cool.

We've had many users ask us about support for doing BE's on systems using GELI full-disk encryption. The issue I currently see with this is that "beadm" doesn't have a way to create / destroy snapshots on a separate boot-pool, if you have your kernel and such not on pool/ROOT/be.

Is this something that is technically feasible? Could we hack in support for setting a BOOTPOOL variable in beadm.conf, and have it just snap / destroy at the same time as the root pool?

Funcionality similar to https://www.illumos.org/issues/7537 should be implemented.

The primary use case is for developers, who develop FreeBSD and create something that can panic. Using nextboot BE, might help them.

From illumos beadm man page:

            -t

               Sets temporary, one time activation. For next boot, the beName
               is used for boot, and the temporary activation is removed.
               When temporary activation is removed, the next boot will use
               zfs dataset specified in boot pool bootfs property.

           -T

               Removes temporary next boot configuration and makes beName the
               active BE on next boot.

When will beadm support multiple ZFS pools? My setup has two pools:

bootdir (with /boot on it)
zroot (with / on it)

Both are encrypted with geli.

Will beadm ever support multiple pools for a ZFS root setup?

On FreeBSD 9.3 release

$ beadm --version
beadm 1.0 2013/10/13

I have this:

# zfs get canmount system/bootenv/default/usr \
                   system/bootenv/10.1-RELEASE/usr
NAME                             PROPERTY  VALUE     SOURCE
system/bootenv/10.1-RELEASE/usr  canmount  off       local
system/bootenv/default/usr       canmount  off       local

After mounting:

# beadm m': beadm mount 10.1-RELEASE /mnt
Mounted successfully on '/mnt'

I notice /mnt/usr does not contain expected contents. Investigation shows a canmount=no filesystem has been mounted:

$ zfs mount | grep system/bootenv/10.1-RELEASE
system/bootenv/10.1-RELEASE     /mnt
system/bootenv/10.1-RELEASE/tmp  /mnt/tmp
system/bootenv/10.1-RELEASE/usr  /mnt/usr
system/bootenv/10.1-RELEASE/usr/local  /mnt/usr/local
system/bootenv/10.1-RELEASE/usr/obj  /mnt/usr/obj
system/bootenv/10.1-RELEASE/usr/src  /mnt/usr/src
system/bootenv/10.1-RELEASE/var  /mnt/var
system/bootenv/10.1-RELEASE/var/audit  /mnt/var/audit
system/bootenv/10.1-RELEASE/var/empty  /mnt/var/empty
system/bootenv/10.1-RELEASE/var/log  /mnt/var/log
system/bootenv/10.1-RELEASE/var/tmp  /mnt/var/tmp

When activating a new BE and rebooting into it, the /boot/entropy file will be from the previous boot. As such, there would be deterministic seeding of the entropy pool. You can create a new /boot/entropy by doing dd if=/dev/random of=/path/to/mounted/be/boot/entropy bs=4096 count=1.

From illumos beadm manpage:

           -o property=value

               Create the datasets for new BE with specific ZFS properties.
               Multiple -o options can be specified.  See zfs(1M) for more
               information on the -o option.

It would be useful, it FreeBSD supported this feature.

Hello. I have started using beadm and it's awesome. I however was confused about the space which beadm shows and wanted to know the reasoning behind it.
Why do we include the size of a zfs file system's origin ( if D is not specified ) with beadm list ? This initially had me confused as it showed that the sum of all the spaces for my boot environment actually exceeds the pool's size but then looking at the code, this is of course intentional.
Looking forward to hearing from you on this, thank you!

Would you consider replacing the hard-coded ROOT dataset with a configurable variable?

I see that there's already a $ROOTFS used in the script.

Thanks.

OS: Void Linux x86_64
Kernel: 5.3.16_1
CPU: Intel i3-8130U (4) @ 3.400GHz

Void Linux, zfs on root

creation, deletion of boot environments work as advertised.

just unable to roll back

i have activated the initial BE

beadm list
BE Active Mountpoint Space Created
initial R / 0.0K
test2 N / 7.9M 2019-12-15 20:29

always starts the "current" BE

Current BE: zroot/ROOT/freebsd83
Activating BE: zroot/ROOT/freebsd90

# zfs get -r -H -o name,value -s local mountpoint zroot/ROOT/freebsd83
zroot/ROOT/freebsd83    legacy
zroot/ROOT/freebsd83/tmp        /tmp
zroot/ROOT/freebsd83/usr        /usr
zroot/ROOT/freebsd83/var        /var

As you see, I created /tmp, /usr and /var datasets in my BE pool. I'm now finding that due to the legacy mountpoint on the root, I'll need to manually move these when activating.

I haven't quite figured out the best way to handle this yet as changing the mountpoint immediately mounts it.

# touch /tmp/8
# zfs set mountpoint=/tmp zroot/ROOT/freebsd90/tmp
# ls -al /tmp/8
ls: /tmp/8: No such file or directory

I might have to just move these to my /etc/fstab, but I'll likely need to add all of their children there as well.

I believe that moving the root to mountpoint=/ would yield the same problem as it would remount the new BE ontop of the current BE before rebooting.

Consider adding a -n flag to dangerous commands such as beadm create, beadm activate, beadm destroy, beadm mount, beadm unmount, beadm rename.

Personally, I'm always scared that something may go terribly wrong, and want to see what will be done first.

Perhaps a -i mode which could simply just run the beadm command with -n, then rerun after confirmation without -n.

I have a zpool on which I have both FreeBSD and Linux. I have separate datasets under FreeBSD for things like /usr, /var, ...

I don't want to set their mountpoints as ZFS properties and use the canmount property, because mounting these does not make sense when I boot into Linux, which has its own /usr. I resorted to using legacy mounts and fstab for everything, because that way FreeBSD and Linux can each have their own fstabs that only mount the correct things.

However, this does not work automatically with beadm, because when creating a new BE, the fstab in the new BE still refers to the datasets from the old BE. I need to manually adjust the fstab, so the right datasets are mounted when I reboot.

It would be nice if beadm could automatically adjust the fstab in the new boot environment by searching it for entries for the old datasets and adjusting them to refer to the new ones.

For example, my old BE dataset name is zfs/FreeBSD/default and contains datasets zfs/FreeBSD/default/usr and others, which have mountpoint=legacy and are listed in fstab. When i do beadm create upgrade, it creates the correct dataset hierarchy under zfs/FreeBSD/upgrade, which is nice, but the fstab still contains zfs/FreeBSD/default/usr rather than the new zfs/FreeBSD/upgrade/usr. It would be nice if beadm could also automatically correct the fstab to refer to the new datasets.

There should really be a test suite for this. Have it create a zpool on file or memory device. Create a jail, delegate the pool to the jail. Using jexec, execute beadm inside the jail, testing various scenarios and setups. Then test the resulting zfs list mount, and checking properties on cloned datasets, ensuring they match expectations. After every test, the jail should be rolled back to a clean starting point.

I'm happy to work on something like this, just documenting here for now for discussion.

Recently I wated to remove obsolete BE's and got asked this question. I'm not sure why I'm being asked this because it statement is false.

odin@asgard ~ % beadm list

BE     Active Mountpoint  Space Created
103    -      -           25.6M 2016-10-11 19:41
103p10 -      -           12.3M 2016-10-11 19:51
11p2   NR     /           17.9G 2016-10-31 23:25


odin@asgard ~ % sudo beadm destroy 103

Are you sure you want to destroy '103'?
This action cannot be undone (y/[n]): y

Boot environment '103' was created from existing snapshot 
Destroy '11p2@2016-10-11-19:51:16' snapshot? (y/[n]): n   <------ What!?

Origin snapshot '11p2@2016-10-11-19:51:16' will be preserved
Destroyed successfully

It;s obviously false claim as there's no way old snapshot can be created from the new snapshot. It was created long time before new one. What does this mean? Is this a bug?

If I create and activate a BE ending with a colon eg. 'TEST:' the environment is not bootable:
the FreeBSD loader gives an error message:
can't load 'kernel'

I am running FreeBSD 11.0-CURRENT.

Regards,
Maurizio

My boot filesystems are in zroot/system/root/{BE1,BE2...}
POOL=zroot and I BEDS=system/root is set in /usr/local/etc/beadm.conf

However, '/' in BEDS results in sed errors and breaks beadm functionality.
What's worse, beadm ignores the errors and proceeds running various commands with root permissions using invalid arguments. Luckily it didn't mess anything up, but it may be prudent to add "set -e" at the beginning of the script to catch cases when things go wrong.

I've worked the issue around by explicitly escaping '/' in beadm.conf:
BEDS='system\/root'

Use case

Before the new beadm, I would sometimes shutdown now, enter a shell then exit.

Enhancement request

In the example below, explicit (longhand) ./beadm reroot n253861-92e6b4712b5-a would have rerooted the environment, which is already NR.

Simpler:

./beadm reroot

– please, can this be implemented? Reroot without specifying a beName.

(Shorthand. Less manual typing, which can be especially helpful in a tty, where copy and paste are not possible, and boot environment names might be long or complex.)

% cd /usr/home/grahamperrin/dev/beadm/
% ./beadm list
BE                    Active Mountpoint  Space Created
n250511-5f73b3338ee-d -      -            4.9G 2021-11-13 15:43
n252381-75d20a5e386-b -      -            6.8G 2022-01-12 23:23
n252450-5efa7281a79-a -      -            6.5G 2022-01-14 19:27
n252483-c8f8299a230-b -      -            4.8G 2022-01-17 14:24
n252505-cc68614da82-a -      -            4.9G 2022-01-18 14:26
n252531-0ce7909cd0b-h -      -            5.7G 2022-02-06 12:24
n252997-b6724f7004c-c -      -            6.2G 2022-02-11 23:07
n253116-39a36707bd3-e -      -            5.7G 2022-02-20 07:03
n253343-9835900cb95-c -      -            1.5G 2022-02-27 14:58
n253627-25375b1415f-e -      -          463.9M 2022-03-12 18:20
n253776-d5ad1713cc3-a -      -          244.0M 2022-03-14 23:40
n253776-d5ad1713cc3-b -      -          631.5M 2022-03-18 09:31
n253861-92e6b4712b5-a NR     /          162.9G 2022-03-19 07:40
% ./beadm reroot
usage:
  beadm activate <beName>
  beadm create [-e nonActiveBe | -e beName@snapshot] <beName>
  beadm create <beName@snapshot>
  beadm destroy [-F] <beName | beName@snapshot>
  beadm export [-v] <beName>
  beadm import [-v] <beName>
  beadm list [-a] [-s] [-D] [-H]
  beadm rename <origBeName> <newBeName>
  beadm mount <beName> [mountpoint]
  beadm { umount | unmount } [-f] <beName>
  beadm reroot <beName>
  beadm version
% 

Thank you

Hi,

I was reading the how-to, and I noticed it was written in April 2012:

https://cdn.rawgit.com/vermaden/beadm/master/HOWTO.htm

I was wondering if anything has changed in regards of preparing filesystem to benefit from beadm while installing FreeBSD 10.2?

From illumos beadm manpage:

   -d description

               Create a new BE with a description associated with it.

This adds description to BE and that description is displayed in the boot loader screen selection menu. This can be handy for end users.

Hello,

I tried to migrate to beadm type structure on 10.0 and am not successful. Before I try it again, I wanted to check and be clear if sysutils/beadm or sysutil/devel-beadm can boot from a geli encrypted drive?

The startup drive is configured as follows:
gpart show ada0
=> 34 1000215149 ada0 GPT (477G)
34 1024 1 freebsd-boot (512K)
1058 990 - free - (495K)
2048 83886080 2 freebsd-swap (40G)
83888128 916327055 3 freebsd-zfs (437G)

The third partition is encrypted and requires a password at boot. Essentially, I could never get beadm to recognize the boot environment on that partition.

Thanks,

Hello,

Doesn't look like beadm works with gawk I'm running gawk from this revision installed from source:
http://git.savannah.gnu.org/cgit/gawk.git/commit/?id=aa4a2c09d9b438e628254238e184ebff2c6b7acc

% beadm list
awk: cmd. line:18: warning: escape sequence `\/' treated as plain `/'
awk: cmd. line:51: fatal: attempt to use scalar `FSNAMES' as an array

% which awk
/usr/local/bin/awk

% awk -V
GNU Awk 4.1.60, API: 2.0
Copyright (C) 1989, 1991-2016 Free Software Foundation.

Is there any csh completion for beadm or indeed bectl?

Hi,

I have a question about how beadm works. In _be_new on line 126 (https://github.com/vermaden/beadm/blob/master/beadm#L110) there is this code:

    # create boot environment from other boot environment
    if zfs list -H -o name ${1}@${2##*/} 1> /dev/null 2> /dev/null
    then
      echo "ERROR: Snapshot '${1}@${2##*/}' already exists"
      exit 1
    fi
    # snapshot format
    FMT=$( date "+%Y-%m-%d-%H:%M:%S" )
    if ! zfs snapshot -r ${1}@${FMT} 1> /dev/null 2> /dev/null
    then
      echo "ERROR: Cannot create snapshot '${1}@${FMT}'"
      exit 1
    fi

So if am on BE default and I want to create a new BE 10.0-RELEASE-p7, the above code will check for the existence of a snapshot called 10.0-RELEASE-p7. I might be wrong, about this, but it looks like it is checking for that snapshot.

I was wondering why beadm was checking for the existence of that snapshot, I don't see that particular snapshot being used.

I ask because I had a BE named default that has a snapshot 10.0-RELEASE-p7 and I got an error trying to create a BE called 10.0-RELEASE-p7 because it said that snapshot already existed, but I can't see where that would be needed, it looks like it creates a snapshot based on the date (in the FMT variable).

Anyway, this could all be wrong, but I was curious. Thanks for beadm, it's been a lifesaver.

1. The manpage is lacking any mention of nicknames.
2. The usage for umount/mount/active are lacking nickname documentation