seamusdemora / ronr-rpi-image-utils

tools to create & update a backup of a running RPiOS system to a raw image file

RonR-RaspberryPi-image-utils

NB scruss is not the author or maintainer of these files (same goes for seamusdemora, who assumed maintenance of this repo from scruss). Please take up any issues or questions in the Image File Utilities thread of the Raspberry Pi Forums site. IOW: This is a file repository only; no support is available here.

Files here are a toolset to create and update a backup of a running RPi OS to a raw image file. The files are copies of those posted on the Raspberry Pi Forums site. The file attachments in that forum don't seem to be persistent (and are subject to other annoyances imposed by CloudFlare). Consequently, this repo was created by user scruss and is now maintained by seamus, to ensure a current working copy of image-utils is always available through git.

An Overview

I've used RonR's image-utils for several years now, and I've become a big fan. Image-utils creates a complete backup of a Raspberry Pi quickly and efficiently; these backups are rendered in the form of an "image file". The *.img format is ideal as a backup because it's a complete backup, it's portable, and it can be loop-mounted. In other words: If your system or SD card or NVME drive becomes corrupted, it can be restored to operation with minimal effort. This restoration requires 3 "ingredients", and about 5 minutes:

1. The *.img file - created by routine/scheduled runs of image-backup on your system
2. A spare micro SD card (or NVME drive)
3. Etcher to write the .img file to the micro-SD card (or NVME drive)

The speed and efficiency of image-backup are especially noteworthy. Because image-backup uses rsync for file copying and syncing, a backup requires only the storage space that is actually used by your system. This is not the same as dd:

1. dd has no way to tell which portions of your drive/SD card are being used versus which portions are not, because dd has no concept of a file. Consequently, a dd backup of a 32 GB SD card requires: ...32GB!!
2. Because of this fundamental limitation, dd is "v-e-r-y s-l-o-w".

By comparison, for my systems (Lite; running headless), a backup of a 32GB SD card requires typically a 3-5GB *.img file, and 5-10 minutes; that includes the time for network transfer to a NAS device.

Another efficiency of image-utils is its ability to update an *.img file. In other words, instead of creating an entire new *.img file from scratch, it can update an existing *.img file to incorporate any changes to the filesystems since the last backup. This ability to update further reduces the time required for a backup from 5-10 minutes to (potentially) seconds.

How Do I Use This Repo?

This repo was created to make a current copy of the RPi image-utils toolset available through git. There are many resources available online describing the use of git, so these instructions are minimal. If you have questions, please consult a tutorial of your own choosing. The instructions below reflect using bash from a Raspberry Pi OS terminal or SSH, and assume that git is installed:

1. clone the repo

$ cd && pwd
/home/pi
$ git clone https://github.com/seamusdemora/RonR-RPi-image-utils.git

which should yield (something like) the following results:

$ git clone https://github.com/seamusdemora/RonR-RPi-image-utils.git
Cloning into 'RonR-RPi-image-utils'...
remote: Enumerating objects: 161, done.
remote: Counting objects: 100% (94/94), done.
remote: Compressing objects: 100% (69/69), done.
remote: Total 161 (delta 59), reused 44 (delta 24), pack-reused 67
Receiving objects: 100% (161/161), 57.62 KiB | 1.92 MiB/s, done.
Resolving deltas: 100% (95/95), done.
$

NOTE THAT a new folder named: 'RonR-RPi-image-utils' has been created

2. take a look around & verify the 'git clone' operation succeeded:

$ ls -la RonR-RPi-image-utils
drwxr-xr-x  2 pi pi  4096 Feb 26 15:29 deprecated
drwxr-xr-x  8 pi pi  4096 Feb 26 15:29 .git
-rw-r--r--  1 pi pi 14084 Feb 26 15:29 image-backup
-rw-r--r--  1 pi pi  1534 Feb 26 15:29 image-check
-rw-r--r--  1 pi pi  3714 Feb 26 15:29 image-chroot
-rw-r--r--  1 pi pi  3399 Feb 26 15:29 image-compare
-rw-r--r--  1 pi pi  3107 Feb 26 15:29 image-info
-rw-r--r--  1 pi pi  1667 Feb 26 15:29 image-mount
-rw-r--r--  1 pi pi  5711 Feb 26 15:29 image-set-partuuid
-rw-r--r--  1 pi pi  4150 Feb 26 15:29 image-shrink
-rw-r--r--  1 pi pi 13740 Feb 26 15:29 README.md
-rw-r--r--  1 pi pi  4086 Feb 26 15:29 README.txt
$

The README.md is this document - the one you're reading now. The README.txt file contains RonR's user's guide for image-utils. The deprecated folder contains an old file discarded by RonR some time ago. The .git folder contains all of the "stuff" that makes git work. And the image-* files are the image-utils files. We'll discuss what to do with the image-utils files below.

3. keep your clone synced to stay current:

Changes to image-utils are infrequent, but they do happen from time to time. You'll want to keep your copies updated to match the latest release. Here's how:

$ cd ~/RonR-RPi-image-utils
$ git config pull.rebase false    # this only needs to be done one time (the first time)
$ git pull                        # all subsequent updates require only this command 

Staging & Usage

Once you've cloned the image-utils files to your local git repo, you'll likely find they are much easier to use by following the very simple install procedure below. Assuming that /usr/local/sbin is in your PATH, using this install procedure makes the utilities easier to use from the command line, or (for example) in a cron job. Here's how to install:

$ cd
$ sudo install --mode=755 ~/RonR-RPi-image-utils/image-* /usr/local/sbin

Creating vs. Updating .img backups

Refer to the Image File Utilities thread of the Raspberry Pi Forums site for documentation & support. The following is offered only as an illustration/example:

Create the .img backup:

To create a NEW image backup, use the sudo image-backup command; you will be prompted for inputs. The ones I typically use are shown below - immediately following the question mark ?:

$ sudo image-backup

Image file to create? /mnt/SynologyNAS/rpi_share/raspberrypi3b/20230212_Pi3B_imagebackup.img

Initial image file ROOT filesystem size (MB) [2317]? 2400

Added space for incremental updates after shrinking (MB) [0]? 200

Create /mnt/SynologyNAS/rpi_share/raspberrypi3b/20230212_Pi3B_imagebackup.img (y/n)?y

This will take a few minutes depending on your model Pi, the size of your file system & other variables. Upon completion, you should find the image file you specified in the location specified in your answer to the first prompt/question above. This image file contains everything exactly as it was in your file system at the time of the backup. This image file may be written to an SD card, or mount-ed as another file system on your RPi (you can use the image-mount utility for this).

Update an existing .img backup:

To update the image file you have created is even easier; sudo image-backup <IMG_TO_UPDT>, or:

$ sudo image-backup /mnt/SynologyNAS/rpi_share/raspberrypi3b/20230212_Pi3B_imagebackup.img

In other words, simply add the URL of the .img file you wish to update to the basic sudo image-backup command.

Bon Voyage

This concludes the README.md file. Once again, any and all questions re image-utils should be submitted to RonR's forum page.

---

ronr-rpi-image-utils's Issues

Error Messages on Bookworm

Hi
I'm trying to use image file utils on Raspberry Pi 3B+ with bookworm. The image file is created, but image-backup ends with "Unable to create backup". But anyhow, the image file is created and can also be mounted (see attached Raspberry CLI output)

Could you have a look at the following output? Do you have an idea what the reason for the error messages could be and how they could be avoided?

pi@raspberrypi:~ $ sudo image-backup

Image file to create? /mnt/SamsungT7/Raspberry/Bookworm_2023_10_23.img

Initial image file ROOT filesystem size (MB) [3057]? 

Added space for incremental updates after shrinking (MB) [0]? 

Create /mnt/SamsungT7/Raspberry/Bookworm_2023_10_23.img (y/n)? y

Starting full backup (for incremental backups, run: /usr/local/sbin/image-backup /mnt/SamsungT7/Raspberry/Bookworm_2023_10_23.img)
mount: (hint) your fstab has been modified, but systemd still uses
       the old version; use 'systemctl daemon-reload' to reload.
mount: (hint) your fstab has been modified, but systemd still uses
       the old version; use 'systemctl daemon-reload' to reload.
rsync: [generator] symlink "/tmp/img-backup-mnt/boot/cmdline.txt" -> "firmware/cmdline.txt" failed: Operation not permitted (1)
rsync: [generator] symlink "/tmp/img-backup-mnt/boot/config.txt" -> "firmware/config.txt" failed: Operation not permitted (1)
rsync: [generator] symlink "/tmp/img-backup-mnt/boot/overlays" -> "firmware/overlays" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]

Unable to create backup

pi@raspberrypi:~ $ 
pi@raspberrypi:~ $ sudo image-check /mnt/SamsungT7/Raspberry/Bookworm_2023_10_23.img

Check /mnt/SamsungT7/Raspberry/Bookworm_2023_10_23.img (y/n)? y

e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop0p2: 78031/195840 files (0.1% non-contiguous), 639638/782336 blocks

tune2fs 1.47.0 (5-Feb-2023)
Filesystem volume name:   <none>
Last mounted on:          /tmp/img-backup-mnt
Filesystem UUID:          0b2fcd94-8a65-440b-9345-d42771eca555
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         unsigned_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              195840
Block count:              782336
Reserved block count:     39116
Overhead clusters:        30970
Free blocks:              142698
Free inodes:              117809
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      381
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8160
Inode blocks per group:   510
Flex block group size:    16
Filesystem created:       Mon Oct 23 12:11:42 2023
Last mount time:          Mon Oct 23 12:11:45 2023
Last write time:          Mon Oct 23 12:15:31 2023
Mount count:              0
Maximum mount count:      -1
Last checked:             Mon Oct 23 12:15:31 2023
Check interval:           0 (<none>)
Lifetime writes:          2524 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      0fa92eb3-b05b-48f6-93ac-49f07fd27c90
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0x58f436b6

pi@raspberrypi:~ $ 
pi@raspberrypi:~ $ sudo image-mount /mnt/SamsungT7/Raspberry/Bookworm_2023_10_23.img /mnt/PiSD

Mount /mnt/SamsungT7/Raspberry/Bookworm_2023_10_23.img on /mnt/PiSD (y/n)? y

e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop0p2: 78031/195840 files (0.1% non-contiguous), 639638/782336 blocks
mount: (hint) your fstab has been modified, but systemd still uses
       the old version; use 'systemctl daemon-reload' to reload.

/mnt/SamsungT7/Raspberry/Bookworm_2023_10_23.img mounted on /mnt/PiSD

When done, run:
umount /mnt/PiSD
losetup -d /dev/loop0

pi@raspberrypi:~ $ 

Figure out a place for issues & discussions

The only place mentioned in the readme for discussions is dead

README file & Documentation

The documentation is very sparse, and is currently formatted as plain text. This makes it difficult to read/use online.

An "automated" solution is needed because I WILL NOT add markup at each revision.

Write failed on `/tmp/..`: No space left on device

I installed Ubuntu on a 256GB microSD card using RPi Imager. I'm trying to port this image to an SSD. I have a USB drive /dev/sdb1 mounted on /mnt/usbdrive (with sudo mount /dev/sdb1 /mnt/usbdrive) so that I can create a backup of the microSD card.

When I run sudo image-backup, I a no space left on device error. These are the complete details:

root@containerberry:~# 
root@containerberry:~# sudo image-backup

Image file to create? /mnt/usbdrive/pibackup/20240217_Pi4B_imagebackup.img

Initial image file ROOT filesystem size (MB) [3245]? 

Added space for incremental updates after shrinking (MB) [0]? 

Create /mnt/usbdrive/pibackup/20240217_Pi4B_imagebackup.img (y/n)? y


Starting full backup (for incremental backups, run: /usr/local/sbin/image-backup /mnt/usbdrive/pibackup/20240217_Pi4B_imagebackup.img)

rsync: [receiver] write failed on "/tmp/img-backup-mnt/usr/lib/firmware/qed/qed_init_values-8.40.33.0.bin": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(380) [receiver=3.2.7]
rsync: [sender] write error: Broken pipe (32)

Unable to create backup

root@containerberry:~# 
root@containerberry:~# 
root@containerberry:~# 
root@containerberry:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           781M  3.1M  778M   1% /run
/dev/mmcblk0p2  235G  3.3G  222G   2% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/mmcblk0p1  253M  143M  110M  57% /boot/firmware
tmpfs           781M  4.0K  781M   1% /run/user/1000
/dev/sdb1       121G   12G  110G  10% /mnt/usbdrive
root@containerberry:~# 
root@containerberry:~# 
root@containerberry:~# 
root@containerberry:~# lsblk --fs
NAME        FSTYPE   FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0       squashfs 4.0                                                          0   100% /snap/core20/2186
loop1       squashfs 4.0                                                          0   100% /snap/lxd/24326
loop2       squashfs 4.0                                                          0   100% /snap/core20/1977
loop3       squashfs 4.0                                                          0   100% /snap/lxd/27038
loop4       squashfs 4.0                                                          0   100% /snap/snapd/19459
loop5       squashfs 4.0                                                          0   100% /snap/snapd/20674
sda                                                                                        
sdb                                                                                        
└─sdb1      exfat    1.0   STRONTIUM12 EE1D-5C11                             109.2G     9% /mnt/usbdrive
mmcblk0                                                                                    
├─mmcblk0p1 vfat     FAT32 system-boot F04B-5543                             109.6M    57% /boot/firmware
└─mmcblk0p2 ext4     1.0   writable    877a65cf-888e-47c6-b4e0-ec5c52c54be8  221.9G     1% /
root@containerberry:~# 
root@containerberry:~# 
root@containerberry:~# 
root@containerberry:~# 
root@containerberry:~# cd /mnt/usbdrive/pibackup/
root@containerberry:/mnt/usbdrive/pibackup# 
root@containerberry:/mnt/usbdrive/pibackup# ll
total 3847424
drwxr-xr-x 2 root root     131072 Feb 17 01:19 ./
drwxr-xr-x 8 root root     131072 Feb 17 01:05 ../
-rwxr-xr-x 1 root root 3939500032 Feb 17 01:45 20240217_Pi4B_imagebackup.img*
root@containerberry:/mnt/usbdrive/pibackup# 
root@containerberry:/mnt/usbdrive/pibackup# 
root@containerberry:/mnt/usbdrive/pibackup# 

I dont understand why there's no space.

Unsupported partition table type:

Tried to run image-utils on a Rasperry PI clone (Bigtreetech Pi v1.2) of my 3D printer. Got this message:
"Unsupported partition table type:"
Note that it is empty after it.

What can I do?
Thanks

Add $MNTPATH to RSYNC

If, like I, you get the message "No more room on device" when creating or updating and image, you probably need to move the TMP directory to another device. The script sets the variable MNTPATH which can be altered to do this however, RSYNC uses the system /TMP by default.
Add the following to the RSYNC command line at line number 99

--temp-dir="$MNTPATH"
In my changed script the line starts:
rsync -aDH --temp-dir="$MNTPATH" --partial ........etc.