Build error: Unable to find free loop device about build

[edumeneses]

Hello everyone, and thanks for this awesome project.

I tested with a Pop!_OS host (Ubuntu 22.04) and found that it cannot find a free loop when using the main branch. It works fine if you checkout to v24.0.2.

Also, I found that building using docker works on the main branch (on my machine) when I modified line 229 in the partitioning.sh file. I changed the old amount (10) to 30:

[[ $FIND_LOOP_CYCLES -gt 30 ]] && exit_with_error "Unable to find free loop device"

A lsblk call in my terminal showed more than 20 loop devices in use (blaming Snap for that). Could the assumption of a free loop under 10 be the reason for the error ?

[AaronNGray]

I found a solution to my issue it appears snaps was causing an issue with eating loop devices on my machine :-

https://ashrafur.medium.com/how-to-clean-up-old-snap-revisions-and-remove-loop-devices-93e38f1ad1f8

[hqnicolas]

First try some officially supported target like generic x86 / arm64 image. tvboxes are untested ...

you have a point! but the V24.02 solve everything

I will start to advice the unnoficial people and the chinese shitboxes to use this:

git clone --depth 1 --branch v24.0.2 https://github.com/armbian/build armbian

[loganmancuso]

Hello everyone, and thanks for this awesome project.

I tested with a Pop!_OS host (Ubuntu 22.04) and found that it cannot find a free loop when using the main branch. It works fine if you checkout to v24.0.2.

Also, I found that building using docker works on the main branch (on my machine) when I modified line 229 in the partitioning.sh file. I changed the old amount (10) to 30:

[[ $FIND_LOOP_CYCLES -gt 30 ]] && exit_with_error "Unable to find free loop device"

A lsblk call in my terminal showed more than 20 loop devices in use (blaming Snap for that). Could the assumption of a free loop under 10 be the reason for the error ?

This worked for me. Snap did not have any dangling loop devices on my laptop but allowing it to continue to look for an available loopback to use allowed my build to work thanks so much. I think a hardcoded limit of 10 might be a bad assumption but im not qualified to say.

[kadrabak]

Great, applied.

[alex3d]

Pre-allocation of 96 loop devices greatly increases chances of success, but this race condition could be prevented altogether by atomic find&take (losetup --show ...). Here is PR #6729

[github-actions]

Jira ticket: AR-2108

[AaronNGray]

deleting the cache directory and rebooting seems to solve it.

[AaronNGray]

Still getting this issue after failed builds :-

https://paste.armbian.com/yarohanuzi

@igorpecovnik - Its like loop devices are not being freed if a build fails !

[AaronNGray]

I will install another version of Ubuntu on a new drive and test whether the problem is specific to this install or not.

This seems to solve the issue temporarily.

How to clean up old Snap revisions and remove loop devices :-

https://ashrafur.medium.com/how-to-clean-up-old-snap-revisions-and-remove-loop-devices-93e38f1ad1f8

[hqnicolas]

Same here, clear cache don't help

$ lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy

$ uname -a
Linux 6.5.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Clean install linux solve....
for 1 build.....

[igorpecovnik]

This loop devices problem is related to host and its kernel. 6.5.x kernel with Jammy should work (better). We have many build runners, some on stock Jammy, some on Armbian Jammy, thus with our kernel. Perhaps try this https://github.com/armbian/sdk ...

/dev/loop devices makes us troubles too, here and there, automation breaks in random manner due to that and which is why we need to repeate few builds here and there.

Also check if you have enough of those loop devices. I have added max_loop=96 to my build machine kernel as otherwise I ran out of them. But due to building several images in parallel.

[igorpecovnik]

Clean install linux solve....
for 1 build.....

It only works once after fresh install and reboot doesn't help??

[hqnicolas]

@igorpecovnik

It only works once after fresh install and reboot doesn't help??

reboot with clean armbian cache doesn't help

i Was trying to build: JP_TVBOX 3566 on V6.6 with jammy 22.04 desktop cinnamon

today I will use the armbian v24.0.2 and so far, so good (first run OK) 15:11 GMT

git clone --depth 1 --branch v24.0.2 https://github.com/armbian/build armbian

[igorpecovnik]

i Was trying to build: JP_TVBOX 3566 on V6.6 with jammy 22.04 desktop cinnamon

First try some officially supported target like generic x86 / arm64 image. tvboxes are untested ...

[MegaArman]

Also having this issue on ubuntu 22.04 jammy, kernel 6.5.0-26-generic . I gave no commandline argument to ./compile.sh. My target was the Le Potato board and I had just loaded a .config that I previously saved.

I saw this loopback error after 2 hours, seemed the kernel compiled OK otherwise.

I will give branch v24.0.2 a try:
git clone --depth 1 --branch=v24.02 https://github.com/armbian/build armbian

Update:
this time (again near the end of process) got "Error 2 occurred in mainshell"

Update (solution):
losetup -a <- see which loopback devices you have
sudo umount /dev/loop# <- unmount loopback device of that # if not needed

I unmounted firefox and the snap store.

For good measure, I also added the line
"options loop max_loop=64"
to /etc/modules-load.d/modules.conf
and rebooted.

I didn't confirm if that had any effect, but it worked. Was able to complete the build. I am using Ubuntu Desktop, perhaps the Ubuntu server image wouldn't have had this issue.

[hqnicolas]

(blaming Snap for that).

Me to!

[AaronNGray]

@igorpecovnik please can you look into @edumeneses solution #6417 (comment) and see if solves this issue so we can close it ?

[Ekmansoft]

So, I get this problem too, so we shouldn't close this ticket until we fix it on master. Just counting the number of loop devices seem to be a bad way to check if armbian has created too many of those. Like previous users, I have 20+ from snap. I'll try and see if the partitioning.sh patch helps on my Ubuntu 23.10 system.

[EvilOlaf]

master

master will no longer receive updates. If the problem is successfully tackled in main this can be closed.

[Ekmansoft]

Lol. I meant the main branch, whatever it's called :-)

I am running on main, and the patch did help me. :-)

[kadrabak]

I had similar issues where deleting the cache folder did not always help.
Once I started using fixed image size (see FIXED_IMAGE_SIZE= parameter) my problem "Unable to find free loop device" disappeared.
However, I like the btrfs filesystem and with this, an operating system needs a separate boot partition.
After some kernel upgrades failed, I learned to set BOOTSIZE parameter like in the example below to get a bigger boot partition.

./compile.sh build BOARD=pine64 BOOTSIZE=300 BRANCH=edge BTRFS_COMPRESSION=zlib BUILD_DESKTOP=no BUILD_MINIMAL=no FIXED_IMAGE_SIZE=7000 INSTALL_HEADERS=yes KERNEL_CONFIGURE=no RELEASE=noble ROOTFS_TYPE=btrfs SEVENZIP=no

[kadrabak]

here is a build test on my setup:

./compile.sh build BOARD=bananapim5 BRANCH=edge BUILD_DESKTOP=no BUILD_MINIMAL=yes KERNEL_CONFIGURE=no RELEASE=bookworm
..failed

./compile.sh build BOARD=bananapim5 BRANCH=edge BUILD_DESKTOP=no BUILD_MINIMAL=yes KERNEL_CONFIGURE=no RELEASE=bookworm FIXED_IMAGE_SIZE=7000
..failed

./compile.sh build BOARD=bananapim5 BRANCH=edge BUILD_DESKTOP=no BUILD_MINIMAL=yes KERNEL_CONFIGURE=no RELEASE=bookworm BOOTSIZE=300 FIXED_IMAGE_SIZE=7000 ROOTFS_TYPE=ext4
... success
note: The system image was built with everything in one partition on filesystem ext4 because Armbian build system knows u-boot is able to boot from ext4 so it will not create boot partition

system: Ubuntu Mantic in Virtualbox, 12880MB Ram, 16Cores, 6.6.29-060629-generic
note: everything is done without reboot

[guoxiaoqiao]

I found a solution to my issue it appears snaps was causing an issue with eating loop devices on my machine :-

https://ashrafur.medium.com/how-to-clean-up-old-snap-revisions-and-remove-loop-devices-93e38f1ad1f8

I use Ubuntu 22.04 and this worked for me, thank you very much.

[tjhexf]

I can't build it in ubuntu 22.04, getting this same issue.

[igorpecovnik]

On "ubuntu 22.04" this is not possible to reproduce. Without logs, there is nothing we can do as we are at "it works on our computer" levels ;) Make sure you are using clean stock Ubuntu install with all updates. This is what our CI runs. If you have some software installed, that can eat out loops ... curse of snapd.

Fedora on the other hand is not officially supported, but its probably lack of loop devices at all or lack of capacity - not sufficient number of them.

[kadrabak]

I run in the same issue again without using FIXED_IMAGE_SIZE=7000 and there is a hint as to why this is happening.

Command to list all loop devices =>
losetup -a | sort
/dev/loop0: [0027]:2897018 (/var/lib/snapd/snaps/bare_5.snap)
/dev/loop10: [0027]:3577273 (/var/lib/snapd/snaps/gtk-common-themes_1534.snap)
/dev/loop1: [0027]:5998653 (/var/lib/snapd/snaps/core20_2264.snap)
/dev/loop11: [0027]:4618189 (/var/lib/snapd/snaps/gtk-common-themes_1535.snap)
/dev/loop12: [0027]:6007698 (/var/lib/snapd/snaps/snapd_21465.snap)
/dev/loop13: [0027]:6079887 (/var/lib/snapd/snaps/snapd_21759.snap)
/dev/loop2: [0027]:6007912 (/var/lib/snapd/snaps/core20_2318.snap)
/dev/loop3: [0027]:5604489 (/var/lib/snapd/snaps/core22_1122.snap)
/dev/loop4: [0027]:5999535 (/var/lib/snapd/snaps/core22_1380.snap)
/dev/loop5: [0027]:5604994 (/var/lib/snapd/snaps/firefox_4090.snap)
/dev/loop6: [0027]:5999688 (/var/lib/snapd/snaps/firefox_4209.snap)
/dev/loop7: [0027]:4834933 (/var/lib/snapd/snaps/gnome-3-38-2004_119.snap)
/dev/loop8: [0027]:5266160 (/var/lib/snapd/snaps/gnome-3-38-2004_143.snap)
/dev/loop9: [0027]:5604818 (/var/lib/snapd/snaps/gnome-42-2204_176.snap)

Command to list an empty one =>
losetup -f
/dev/loop14

Guessing one loop device is quite problem when armbian build script needs at least 2, one for boot and second for root filesystem.

[kadrabak]

[🌱] Enabling command-not-found after build-time [ tinkerboard-2:noble-current ]
[🌱] Unmounting [ /data/armbian/build/.tmp/rootfs-d8196a9c-8447-477f-9876-d98ca78c957d ]
[🌿] Actual rootfs size [ 2752MiB ]
[🚸] Rootfs post-tweaks size is larger than estimated tmpfs size [ 2752MiB > 2300MiB ]
[🌱] Preparing image file for rootfs [ tinkerboard-2 noble ]
[🌱] Current rootfs size [ 2752 MiB ]
[🌱] Using user-defined image size [ 7000 MiB ]
[🌱] Creating blank image for rootfs [ truncate: 7000 MiB ]
[🌱] Creating partitions [ /boot: ext4 root: btrfs ]
[🔨] Checking that no-one is using this disk right now ... OK
[🔨]
[🔨] Disk /data/armbian/build/.tmp/rootfs-d8196a9c-8447-477f-9876-d98ca78c957d.raw: 6.84 GiB, 7340032000 bytes, 14336000 sectors
[🔨] Units: sectors of 1 * 512 = 512 bytes
[🔨] Sector size (logical/physical): 512 bytes / 512 bytes
[🔨] I/O size (minimum/optimal): 512 bytes / 512 bytes
[🔨]
[🔨] >>> Script header accepted.
[🔨] >>> Created a new DOS (MBR) disklabel with disk identifier 0xc7059d71.
[🔨] /data/armbian/build/.tmp/rootfs-d8196a9c-8447-477f-9876-d98ca78c957d.raw1: Created a new partition 1 of type 'Linux extended boot' and of size 300 MiB.
[🔨] /data/armbian/build/.tmp/rootfs-d8196a9c-8447-477f-9876-d98ca78c957d.raw2: Created a new partition 2 of type 'Linux' and of size 6.5 GiB.
[🔨] /data/armbian/build/.tmp/rootfs-d8196a9c-8447-477f-9876-d98ca78c957d.raw3: Done.
[🔨]
[🔨] New situation:
[🔨] Disklabel type: dos
[🔨] Disk identifier: 0xc7059d71
[🔨]
[🔨] Device Boot Start End Sectors Size Id Type
[🔨] /data/armbian/build/.tmp/rootfs-d8196a9c-8447-477f-9876-d98ca78c957d.raw1 32768 647167 614400 300M ea Linux extended boot
[🔨] /data/armbian/build/.tmp/rootfs-d8196a9c-8447-477f-9876-d98ca78c957d.raw2 647168 14335999 13688832 6.5G 83 Linux
[🔨]
[🔨] The partition table has been altered.
[🔨] Syncing disks.
[💥] error! [ Unable to find free loop device ]
[💥] Exiting with error 43 [ at /data/armbian/build/lib/functions/logging/traps.sh:1
exit_with_error() --> lib/functions/logging/traps.sh:1
prepare_partitions() --> lib/functions/image/partitioning.sh:229
do_with_logging() --> lib/functions/logging/section-logging.sh:81
build_rootfs_and_image() --> lib/functions/main/rootfs-image.sh:86
full_build_packages_rootfs_and_image() --> lib/functions/main/default-build.sh:36
do_with_default_build() --> lib/functions/main/default-build.sh:42
cli_standard_build_run() --> lib/functions/cli/cli-build.sh:25
armbian_cli_run_command() --> lib/functions/cli/utils-cli.sh:136
cli_entrypoint() --> lib/functions/cli/entrypoint.sh:176
main() --> compile.sh:50
]
[💥] Cleaning up [ please wait for cleanups to finish ]
[🌿] Unmounting recursively [ SDCARD - be patient ]
[🌿] Unmounting recursively [ MOUNT - be patient ]
[🚸] Freeing loop [ trap_handler_cleanup_rootfs_and_image /dev/loop13 ]
[🌿] Freeing loop device [ /dev/loop13 ]
[🌿] ANSI log file built; inspect it by running: [ less -RS output/logs/log-build-d8196a9c-8447-477f-9876-d98ca78c957d.log.ans ]
[🌿] Share log manually (or SHARE_LOG=yes): [ curl --data-binary @output/logs/log-build-d8196a9c-8447-477f-9876-d98ca78c957d.log.ans https://paste.armbian.com/log ]

[kadrabak]

log says freeing Freeing loop device [ /dev/loop13 ]

after finishing all this I run these commands:
$ losetup -f
/dev/loop14

$ losetup -a
/dev/loop1: []: (/var/lib/snapd/snaps/core20_2264.snap)
/dev/loop8: []: (/var/lib/snapd/snaps/gnome-3-38-2004_119.snap)
/dev/loop6: []: (/var/lib/snapd/snaps/firefox_4209.snap)
/dev/loop13: []: (/var/lib/snapd/snaps/snapd_21759.snap)
/dev/loop4: []: (/var/lib/snapd/snaps/core22_1380.snap)
/dev/loop11: []: (/var/lib/snapd/snaps/gtk-common-themes_1535.snap)
/dev/loop2: []: (/var/lib/snapd/snaps/core20_2318.snap)
/dev/loop0: []: (/var/lib/snapd/snaps/bare_5.snap)
/dev/loop9: []: (/var/lib/snapd/snaps/gnome-42-2204_176.snap)
/dev/loop7: []: (/var/lib/snapd/snaps/gnome-3-38-2004_143.snap)
/dev/loop5: []: (/var/lib/snapd/snaps/firefox_4090.snap)
/dev/loop12: []: (/var/lib/snapd/snaps/snapd_21465.snap)
/dev/loop3: []: (/var/lib/snapd/snaps/core22_1122.snap)
/dev/loop10: []: (/var/lib/snapd/snaps/gtk-common-themes_1534.snap)

So for me device 13 was taken already because when build script finished this loop device is taken with snapd.
Also and there is free device 14
???
Cannot understand it

[igorpecovnik]

Cannot understand it

Evil of snap :(

Many of us are using snapd free Armbian Ubuntu where this problem doesn't happen. For machines, where we build in parallel, I had to add max_loop to kernel command line:

GRUB_CMDLINE_LINUX_DEFAULT=" console=tty1 max_loop=96"

ls -l /dev/loop
loop0#         loop18#        loop29#        loop40#        loop51#        loop63p1#      loop73#        loop87#      
loop0p1#       loop18p1#      loop3#         loop41#        loop52#        loop64#        loop74#        loop87p1#    
loop0p2#       loop19#        loop30#        loop41p1#      loop53#        loop64p1#      loop75#        loop88#      
loop1#         loop2#         loop31#        loop42#        loop54#        loop65#        loop76#        loop88p1#    
loop10#        loop20#        loop31p1#      loop43#        loop55#        loop66#        loop77#        loop89#      
loop11#        loop21#        loop32#        loop44#        loop56#        loop67#        loop78#        loop9#       
loop12#        loop22#        loop33#        loop45#        loop57#        loop67p1#      loop79#        loop90#      
loop13#        loop23#        loop34#        loop46#        loop58#        loop67p2#      loop8#         loop90p1#    
loop14#        loop24#        loop35#        loop47#        loop59#        loop68#        loop80#        loop91#      
loop14p1#      loop25#        loop36#        loop48#        loop6#         loop69#        loop81#        loop92#      
loop15#        loop26#        loop37#        loop49#        loop60#        loop7#         loop82#        loop93#      
loop15p1#      loop26p1#      loop37p1#      loop49p1#      loop60p1#      loop70#        loop83#        loop94#      
loop16#        loop26p2#      loop38#        loop5#         loop61#        loop70p1#      loop84#        loop95#      
loop17#        loop27#        loop39#        loop50#        loop62#        loop71#        loop85#        loop-control%
loop17p1#      loop28#        loop4#         loop50p1#      loop63#        loop72#        loop86#

Here even snap can't hurt ;)