crun: bpf create ``: Invalid argument about crun

[giuseppe]

do you have CONFIG_CGROUP_BPF=y in your kernel .config?

[IngwiePhoenix]

Thank you, that was in fact the missing piece! ... and also making CONFIG_OVERLAY_FS a builtin instead.

However, why did Podman - well, crun/conmon to be specific - pick fuse-overlay instead of just giving me an error about the kernel native driver not working? Here is what I had:

# docker run --log-level debug -it riscv64/hello-world
INFO[0000] /usr/bin/podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(/usr/bin/podman run --log-level debug -it riscv64/hello-world)
DEBU[0000] Using conmon: "/usr/bin/conmon"
INFO[0000] Using sqlite as database backend
DEBU[0000] Using graph driver
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] overlay: storage already configured with a mount-program
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
INFO[0000] [graphdriver] using prior storage driver: overlay
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 13
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image riscv64/hello-world (policy: missing)
DEBU[0000] Looking up image "riscv64/hello-world" in local containers storage
DEBU[0000] Normalized platform linux/riscv64 to {riscv64 linux  [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/shortnames.conf"
DEBU[0000] Trying "localhost/riscv64/hello-world:latest" ...
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/run/containers/storage]localhost/riscv64/hello-world:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/riscv64/hello-world:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage]@ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684"
DEBU[0000] Found image "riscv64/hello-world" as "docker.io/riscv64/hello-world:latest" in local containers storage
DEBU[0000] Found image "riscv64/hello-world" as "docker.io/riscv64/hello-world:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage]@ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684)
DEBU[0000] exporting opaque data as blob "sha256:ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684"
DEBU[0000] Looking up image "docker.io/riscv64/hello-world:latest" in local containers storage
DEBU[0000] Normalized platform linux/riscv64 to {riscv64 linux  [] }
DEBU[0000] Trying "docker.io/riscv64/hello-world:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage]@ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684"
DEBU[0000] Found image "docker.io/riscv64/hello-world:latest" as "docker.io/riscv64/hello-world:latest" in local containers storage
DEBU[0000] Found image "docker.io/riscv64/hello-world:latest" as "docker.io/riscv64/hello-world:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage]@ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684)
DEBU[0000] exporting opaque data as blob "sha256:ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684"
DEBU[0000] Looking up image "riscv64/hello-world" in local containers storage
DEBU[0000] Normalized platform linux/riscv64 to {riscv64 linux  [] }
DEBU[0000] Trying "localhost/riscv64/hello-world:latest" ...
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/run/containers/storage]localhost/riscv64/hello-world:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/riscv64/hello-world:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage]@ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684"
DEBU[0000] Found image "riscv64/hello-world" as "docker.io/riscv64/hello-world:latest" in local containers storage
DEBU[0000] Found image "riscv64/hello-world" as "docker.io/riscv64/hello-world:latest" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage]@ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684)
DEBU[0000] exporting opaque data as blob "sha256:ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684"
DEBU[0000] Inspecting image ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684
DEBU[0000] exporting opaque data as blob "sha256:ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684"
DEBU[0000] Inspecting image ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684
DEBU[0000] Inspecting image ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684
DEBU[0000] Inspecting image ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json"
DEBU[0000] Allocated lock 15 for container 99e512aeeba56c31666e2223135d30cd65fa9d17b66b85e05e6be7031b9d9712
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage]@ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684"
DEBU[0000] exporting opaque data as blob "sha256:ef56fe44c1ae4bebc38df8c301c8c5e8ab561b36dae0c734d1018f6c0e1b4684"
DEBU[0000] Created container "99e512aeeba56c31666e2223135d30cd65fa9d17b66b85e05e6be7031b9d9712"
DEBU[0000] Container "99e512aeeba56c31666e2223135d30cd65fa9d17b66b85e05e6be7031b9d9712" has work directory "/var/lib/containers/storage/overlay-containers/99e512aeeba56c31666e2223135d30cd65fa9d17b66b85e05e6be7031b9d9712/userdata"
DEBU[0000] Container "99e512aeeba56c31666e2223135d30cd65fa9d17b66b85e05e6be7031b9d9712" has run directory "/run/containers/storage/overlay-containers/99e512aeeba56c31666e2223135d30cd65fa9d17b66b85e05e6be7031b9d9712/userdata"
DEBU[0000] Handling terminal attach
INFO[0000] Received shutdown.Stop(), terminating!        PID=2451
DEBU[0000] Enabling signal proxying
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/FJWB6J5MCP3JY5MLSOQFJAF7WZ,upperdir=/var/lib/containers/storage/overlay/a79ca8d4e824be1b790b76558138cc9c5c51a9033c6d48c660203a1863da35ba/diff,workdir=/var/lib/containers/storage/overlay/a79ca8d4e824be1b790b76558138cc9c5c51a9033c6d48c660203a1863da35ba/work
ERRO[0000] Unmounting /var/lib/containers/storage/overlay/a79ca8d4e824be1b790b76558138cc9c5c51a9033c6d48c660203a1863da35ba/merged: invalid argument
DEBU[0000] Failed to mount container "99e512aeeba56c31666e2223135d30cd65fa9d17b66b85e05e6be7031b9d9712": creating overlay mount to /var/lib/containers/storage/overlay/a79ca8d4e824be1b790b76558138cc9c5c51a9033c6d48c660203a1863da35ba/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/FJWB6J5MCP3JY5MLSOQFJAF7WZ,upperdir=/var/lib/containers/storage/overlay/a79ca8d4e824be1b790b76558138cc9c5c51a9033c6d48c660203a1863da35ba/diff,workdir=/var/lib/containers/storage/overlay/a79ca8d4e824be1b790b76558138cc9c5c51a9033c6d48c660203a1863da35ba/work": using mount program /usr/bin/fuse-overlayfs: unknown argument ignored: lazytime
fuse: mount failed: Invalid argument
fuse-overlayfs: cannot mount: Invalid argument: exit status 1
DEBU[0000] Made network namespace at /run/netns/netns-a1ab7260-2a09-bf65-cca2-c6829c46f125 for container 99e512aeeba56c31666e2223135d30cd65fa9d17b66b85e05e6be7031b9d9712
[DEBUG netavark::network::validation] "Validating network namespace..."
[DEBUG netavark::commands::setup] "Setting up..."
[INFO  netavark::firewall] Using iptables firewall driver
[DEBUG netavark::network::bridge] Setup network podman
[DEBUG netavark::network::bridge] Container interface name: eth0 with IP addresses [10.88.0.6/16]
[DEBUG netavark::network::bridge] Bridge name: podman0 with IP addresses [10.88.0.1/16]
[DEBUG netavark::network::core_utils] Setting sysctl value for net.ipv4.ip_forward to 1
[DEBUG netavark::network::core_utils] Setting sysctl value for /proc/sys/net/ipv6/conf/eth0/autoconf to 0
[INFO  netavark::network::netlink] Adding route (dest: 0.0.0.0/0 ,gw: 10.88.0.1, metric 100)
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-1D8721804F16F created on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK_FORWARD exists on table filter
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK_FORWARD exists on table filter
[DEBUG netavark::firewall::varktables::helpers] rule -d 10.88.0.0/16 -j ACCEPT exists on table nat and chain NETAVARK-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule -d 10.88.0.0/16 -j ACCEPT created on table nat and chain NETAVARK-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule ! -d 224.0.0.0/4 -j MASQUERADE exists on table nat and chain NETAVARK-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule ! -d 224.0.0.0/4 -j MASQUERADE created on table nat and chain NETAVARK-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule -s 10.88.0.0/16 -j NETAVARK-1D8721804F16F exists on table nat and chain POSTROUTING
[DEBUG netavark::firewall::varktables::helpers] rule -s 10.88.0.0/16 -j NETAVARK-1D8721804F16F created on table nat and chain POSTROUTING
[DEBUG netavark::firewall::varktables::helpers] rule -d 10.88.0.0/16 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT exists on table filter and chain NETAVARK_FORWARD
[DEBUG netavark::firewall::varktables::helpers] rule -d 10.88.0.0/16 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT created on table filter and chain NETAVARK_FORWARD
[DEBUG netavark::firewall::varktables::helpers] rule -s 10.88.0.0/16 -j ACCEPT exists on table filter and chain NETAVARK_FORWARD
[DEBUG netavark::firewall::varktables::helpers] rule -s 10.88.0.0/16 -j ACCEPT created on table filter and chain NETAVARK_FORWARD
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-SETMARK exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-SETMARK exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-MASQ exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-MASQ exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-DNAT exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-DNAT exists on table nat
[INFO  netavark::commands::setup] dns disabled because aardvark-dns path does not exists
[DEBUG netavark::commands::setup] {
  "podman": StatusBlock {
    dns_search_domains: Some(
      [],
      dns_server_ips: Some(
        [],
      ),
      interfaces: Some(
        {
          "eth0": NetInterface {
            mac_address: "8e:d6:44:11:8b:27",
            subnets: Some(
              [
              NetAddress {
                gateway: Some(
                  10.88.0.1,
                ),
                ipnet: 10.88.0.6/16,
              },
            ],
          ),
        },
      },
    ),
  },
}
[DEBUG netavark::commands::setup] "Setup complete"
DEBU[0000] Tearing down network namespace at /run/netns/netns-a1ab7260-2a09-bf65-cca2-c6829c46f125 for container 99e512aeeba56c31666e2223135d30cd65fa9d17b66b85e05e6be7031b9d9712
[DEBUG netavark::commands::teardown] "Tearing down.."
[INFO  netavark::firewall] Using iptables firewall driver
[INFO  netavark::network::bridge] removing bridge podman0
[DEBUG netavark::commands::teardown] "Teardown complete"
DEBU[0000] Cleaning up container 99e512aeeba56c31666e2223135d30cd65fa9d17b66b85e05e6be7031b9d9712
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Container 99e512aeeba56c31666e2223135d30cd65fa9d17b66b85e05e6be7031b9d9712 storage is already unmounted, skipping...
DEBU[0000] ExitCode msg: "mounting storage for container 99e512aeeba56c31666e2223135d30cd65fa9d17b66b85e05e6be7031b9d9712: creating overlay mount to /var/lib/containers/storage/overlay/a79ca8d4e824be1b790b76558138cc9c5c51a9033c6d48c660203a1863da35ba/merged, mount_data=\"lowerdir=/var/lib/containers/storage/overlay/l/fjwb6j5mcp3jy5mlsoqfjaf7wz,upperdir=/var/lib/containers/storage/overlay/a79ca8d4e824be1b790b76558138cc9c5c51a9033c6d48c660203a1863da35ba/diff,workdir=/var/lib/containers/storage/overlay/a79ca8d4e824be1b790b76558138cc9c5c51a9033c6d48c660203a1863da35ba/work\": using mount program /usr/bin/fuse-overlayfs: unknown argument ignored: lazytime\nfuse: mount failed: invalid argument\nfuse-overlayfs: cannot mount: invalid argument\n: exit status 1"
Error: mounting storage for container 99e512aeeba56c31666e2223135d30cd65fa9d17b66b85e05e6be7031b9d9712: creating overlay mount to /var/lib/containers/storage/overlay/a79ca8d4e824be1b790b76558138cc9c5c51a9033c6d48c660203a1863da35ba/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/FJWB6J5MCP3JY5MLSOQFJAF7WZ,upperdir=/var/lib/containers/storage/overlay/a79ca8d4e824be1b790b76558138cc9c5c51a9033c6d48c660203a1863da35ba/diff,workdir=/var/lib/containers/storage/overlay/a79ca8d4e824be1b790b76558138cc9c5c51a9033c6d48c660203a1863da35ba/work": using mount program /usr/bin/fuse-overlayfs: unknown argument ignored: lazytime
fuse: mount failed: Invalid argument
fuse-overlayfs: cannot mount: Invalid argument: exit status 1
DEBU[0000] Shutting down engines

I then verified that fuse and overlay were loaded via lsmod and apt purge'd fuse-overlayfs off my system. And then proceeded to rebuild the kernel. After that, it worked!

Just for completeness sake, in case anyone ever stumbles across this, I am leaving these tidbits here that I stumbled across:

Loaded kernel modules AFTER removing fuse-overlayfs and BEFORE rebooting into the newly built kernel:

# lsmod
Module                  Size  Used by
overlay               200704  0
rfkill                 28672  1
rc_pinnacle_pctv_hd    12288  0
em28xx_rc              24576  0
rc_core                57344  4 em28xx_rc,rc_pinnacle_pctv_hd
bridge                409600  0
a8293                  12288  1
stp                    12288  1 bridge
ts2020                 20480  1
m88ds3103              28672  1
si2157                 20480  1
si2168                 20480  1
firmware_class         16384  3 si2157,m88ds3103,si2168
em28xx_dvb             49152  8
cp210x                 28672  0
em28xx                114688  2 em28xx_rc,em28xx_dvb
tveeprom               20480  1 em28xx
nls_iso8859_1          12288  1
uio_pdrv_genirq        12288  0
fuse                  204800  1

This is log output AFTER having removed fuse-overlayfs. Now I actually got the error, hinting me towards my kernel option being off.

# docker run --log-level debug -it riscv64/hello-world
INFO[0000] /usr/bin/podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(/usr/bin/podman run --log-level debug -it riscv64/hello-world)
DEBU[0000] Using conmon: "/usr/bin/conmon"
INFO[0000] Using sqlite as database backend
DEBU[0000] Using graph driver
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] overlay: storage already configured with a mount-program
ERRO[0000] overlay: removing an item from the merged directory failed: open /var/lib/containers/storage/overlay/compat964775770/merged: no such device or address
ERRO[0008] [graphdriver] prior storage driver overlay failed: kernel does not support overlay fs: kernel returned open /var/lib/containers/storage/overlay/compat964775770/merged: no such device or address when we tried to delete an item in the merged directory: driver not supported
Error: kernel does not support overlay fs: kernel returned open /var/lib/containers/storage/overlay/compat964775770/merged: no such device or address when we tried to delete an item in the merged directory: driver not supported
DEBU[0008] Shutting down engines

Little bit odd, but it worked out. :) I had briefly looked into placing a global storage.conf into /etc/containers, but that didn't change anything. In fact, it got worse, since I have zero experience with the contents of that file...

Anyway, thank you for the pointer - and as said, I am just leaving these bits here, inc ase someone googles something of this sort, in hopes that it will help them. :)

Have a nice day!