mdaffin / loopdev Goto Github PK

Hello @mdaffin

can you please give an update on the maintenance state of this crate? A couple of people depend on loopdev and are waiting on 0.5.0 as to be read in #62 .

Thanks,

@flxo

Running cargo build produces an error, is there some setup step I'm missing or can this crate not be built on OSX?

Error:

$ cargo build 
   Compiling loopdev v0.5.0 (/Users/rethy/rust/loopdev)
error: failed to run custom build command for `loopdev v0.5.0 (/Users/rethy/rust/loopdev)`

Caused by:
  process didn't exit successfully: `/Users/rethy/rust/loopdev/target/debug/build/loopdev-7d51fc8cf50a1a5a/build-script-build` (exit status: 101)
  --- stderr
  /Users/rethy/rust/loopdev/wrapper.h:1:10: fatal error: 'linux/loop.h' file not found
  thread 'main' panicked at 'Could not generate bindings: ClangDiagnostic("/Users/rethy/rust/loopdev/wrapper.h:1:10: fatal error: 'linux/loop.h' file not found\n")', build.rs:11:10
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

System info:

OS: macOS 11.4 20F71 x86_64 
Host: MacBookPro12,1 
Kernel: 20.5.0 
CPU: Intel i7-5557U (4) @ 3.10GHz 
GPU: Intel Iris Graphics 6100 

Just a summary from #27:

The integration tests attach and detach loop devices. This is privileged operation that requires super user rights (or capabilities).
The tests are currently invoked in the CI with sudo to obtain those rights. Executing the tests during development with sudo is cumbersome because it leaves root owned files behind in the target tree.

An attempt to configure a runner that executes the tests with sudo in #27 solves the issue partially: Executing the tests works fine. But cargo doesn't make use of the runner when executing the doc tests which need the same permissions as the integration tests though.

Hi,

we are getting a test failure on arm64 in ubuntu mantic with rust-loopdev 0.4.0-3[1]

864s ---- detach_a_backing_file_default stdout ----
864s thread 'detach_a_backing_file_default' panicked at 'assertion failed: `(left == right)`
864s left: `0`,
864s right: `1`: there should be no loopback devices mounted', tests/integration_test.rs:176:5

I added some print statements, ran it in an arm64 vm (which has normally 3 loop devices used already) and it looks like the loop device count is changing under our feet:

---- detach_a_backing_file_with_offset stdout ----
XXX num_devices_at_start=4
XXX list_device(None).len()=4
XXX start list_device(None): [LoopDeviceOutput { name: "/dev/loop1", size_limit: Some(0), offset: Some(0), back_file: Some("/var/lib/snapd/snaps/lxd_25116.snap") }, LoopDeviceOutput { name: "/dev/loop2", size_limit: Some(0), offset: Some(0), back_file: Some("/var/lib/snapd/snaps/snapd_19459.snap") }, LoopDeviceOutput { name: "/dev/loop0", size_limit: Some(0), offset: Some(0), back_file: Some("/var/lib/snapd/snaps/core22_821.snap") }], len=3, var=4
XXX end list_device(None): [LoopDeviceOutput { name: "/dev/loop1", size_limit: Some(0), offset: Some(0), back_file: Some("/var/lib/snapd/snaps/lxd_25116.snap") }, LoopDeviceOutput { name: "/dev/loop2", size_limit: Some(0), offset: Some(0), back_file: Some("/var/lib/snapd/snaps/snapd_19459.snap") }, LoopDeviceOutput { name: "/dev/loop0", size_limit: Some(0), offset: Some(0), back_file: Some("/var/lib/snapd/snaps/core22_821.snap") }], num_devices_at_start=4
thread 'detach_a_backing_file_with_offset' panicked at 'assertion failed: `(left == right)`
  left: `3`,
 right: `4`: there should be no loopback devices mounted', tests/integration_test.rs:180:5

This is the changed function:

fn detach_a_backing_file(offset: u64, sizelimit: u64, file_size: i64) {
    let num_devices_at_start = list_device(None).len();
    println!("XXX num_devices_at_start={}",num_devices_at_start);
    println!("XXX list_device(None).len()={}", list_device(None).len());
    let _lock = setup();

    println!("XXX start list_device(None): {:?}, len={}, var={}", list_device(None), list_device(None).len(), num_devices_at_start);
    {
        let file = create_backing_file(file_size);
        attach_file(
            "/dev/loop3",
            file.to_path_buf().to_str().unwrap(),
            offset,
            sizelimit,
        );

        let ld0 = LoopDevice::open("/dev/loop3")
            .expect("should be able to open the created loopback device");

        ld0.detach()
            .expect("should not error detaching the backing file from the loopdev");

        file.close().expect("should delete the temp backing file");
    };

    std::thread::sleep(std::time::Duration::from_millis(500));

    println!("XXX end list_device(None): {:?}, num_devices_at_start={}", list_device(None), num_devices_at_start);
    assert_eq!(
        list_device(None).len(),
        num_devices_at_start,
        "there should be no loopback devices mounted"
    );
    detach_all();
}

Looks like list_device(None) right after the setup() call has one less loop device already. Which one? I don't know, because if I print list_device(None) right at the start of the test, then the failure doesn't happen anymore ;)

I know this is not the latest version of rust-loopdev, and I checked the git log to see if anything could be a fix for this, but my rust knowledge is close to zero.

Here is a full test run showing the failure[2].

In my test vm, I have to run the tests multiple times (but not many) until I see the failure. It doesn't matter if it has one or two cpus.

1. https://code.launchpad.net/~git-ubuntu-import/ubuntu/+source/rust-loopdev/+git/rust-loopdev/+ref/applied/ubuntu/devel:
2. https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/arm64/r/rust-loopdev/20230814_100137_214a4@/log.gz

if you build for riscv64gc you get an error from bindgen:

 error: unknown target triple 'riscv64gc-unknown-linux-gnu', please use -triple or -arch
  thread 'main' panicked at 'libclang error; possible causes include:
  - Invalid flag syntax
  - Unrecognized flags
  - Invalid flag arguments
  - File I/O errors
  - Host vs. target architecture mismatch

It has been deprecated: announcement.

Looks like this is blocked by docopt/docopt.rs#128.

The integration tests make use of the losetup utility to attach and delete loop devices. losetup doesn't do any retries upon EBSY or EAGAIN errors that are issues from time to time from the loopdev driver. Especially when running the tests in parallel (which is the cargo default) the tests fail from time to time in the Travis setup. The test implementation tries to workaround this by adding delays before and after losetup calls.
The attempt to migrate the CI to Github actions has the same issues. Looks like the errors happen even more often than on Travis.

A possible solutions is to add retries to the losetup invocations. This is common approach that is seen in almost every piece of code that interacts with the loopback driver.

This would be a bit of a help for Fedora packaging. We're accumulating patches just a bit, but it's rather frowned on in Fedora to pull from anything other than an actual release.

I use fedora 38 (aarch64). Get error output:

error: failed to run custom build command for `loopdev v0.4.0`

Caused by:
  process didn't exit successfully: `/home/qinka/WorkSpaces/mcdcat/target/debug/build/loopdev-e5fd15abe1760b06/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at '"enum_(unnamed_at_/usr/include/linux/loop_h_16_1)" is not a valid Ident', /home/qinka/.cargo/registry/src/mirrors.tuna.tsinghua.edu.cn-2eab394af869c8a2/proc-macro2-1.0.56/src/fallback.rs:811:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

uname -a output:

$ uname -a
Linux etvp-rpi 6.2.11-300.fc38.aarch64 #1 SMP PREEMPT_DYNAMIC Thu Apr 13 20:30:09 UTC 2023 aarch64 GNU/Linux

The new functionality was added in 0.2.2 : This commit introduces LOOP_SET_DIRECT_IO support. However, this IOCTL is absent on distros with 3.x kernels, like CentOS 7. Thus, we have a compilation error:

error[E0432]: unresolved import bindings::LOOP_SET_DIRECT_IO
--> /home/xxxxxxxxx/.cargo/registry/src/github.com-xxxxxxxxx/loopdev-0.2.2/src/lib.rs:40:69
|
40 | loop_info64, LOOP_CLR_FD, LOOP_CTL_GET_FREE, LOOP_SET_CAPACITY, LOOP_SET_DIRECT_IO,
| ^^^^^^^^^^^^^^^^^^ no LOOP_SET_DIRECT_IO in bindings

I suggest to add cargo-feature for such direct io functionality and to mark the function pub fn set_direct_io(&self, direct_io: bool) -> io::Result<()> { with such feature to compile it conditionally.

Seems there is a new way to interact with loop devices, in particular private loop devices via mounting loopfs to some location. From a quick glance it looks like you can have different locations for loop-control and loop* devices other than /dev. Maybe this is what android is doing under the hood? We might need a more generic approach then the swapping out the LOOP_PREFIX depending on the target.

https://lwn.net/Articles/819625/

@mdaffin Would you mind releasing v0.2.2 on crates io?

The new features are:

• Build and run on Android
• Add LoopDevice::major and LoopDevice::minor
• Add readonly, autoclear and direct io flags
• Use bindgen instead of hard coded constants and types

Thanks!