Failed running docker container inside the minimal container about podman-static

[awidjaja]

@mgoltzsche , I tried running redhat's fedora version of the podman container on the same WSL2 distribution (based on your normal podman image), and it works. Maybe there's some configuration that can be followed?

sudo podman run --user podman --privileged quay.io/podman/stable podman run ubi8 echo hello

time="2022-06-25T14:59:44Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:1b890c73c3cf60b04334fded9e3edc647d64dd39ffd078317e2bd69552a2fd1d
Copying blob sha256:1b890c73c3cf60b04334fded9e3edc647d64dd39ffd078317e2bd69552a2fd1d
Copying blob sha256:de63ba066b7c0c23e2434efebcda7800d50d60f33803af9c500f75a69fb76ffa
Copying blob sha256:de63ba066b7c0c23e2434efebcda7800d50d60f33803af9c500f75a69fb76ffa
Copying config sha256:88c8870492b3eb59d6c272244e0f5e8e924450e05216546ad710d276b9e6e2d3
Writing manifest to image destination
Storing signatures
hello

[mgoltzsche]

That's odd. Looks like the container file system wasn't mounted (due to a silently ignored error maybe).
Does it work when you run the command as root (sudo podman ...)?

[awidjaja]

I run the above as root user in the WSL2 host.

[mgoltzsche]

Okay, I haven't used WSL2 yet.
Does the problem also occur when you use the normal image (mgoltzsche/podman:4.1.0, uses runc) instead of the minimal image?
I don't see many differences to my podman info other than that the uidmap/gidmap is null in your case.
You could try to configure /etc/subuid//etc/subgid on your host as described here and see whether that fixes the error.

... Also, I just published a new image mgoltzsche/podman:4.1.1 that you could try.
However on my machine I just realize that, using the root user and podman for the outer container, the nested container's image cannot be pulled due to a network issue - as unprivileged user it works as expected though.

[awidjaja]

It is correct on the host.

cat /etc/subuid
podman:100000:65536

cat /etc/subgid
podman:100000:65536

Using the normal image

`podman run --privileged -u podman:podman mgoltzsche/podman docker run alpine:latest echo hello from nested container

Resolved "mgoltzsche/podman" as an alias (/var/cache/containers/short-name-aliases.conf)
Trying to pull docker.io/mgoltzsche/podman:latest...
Getting image source signatures
Copying blob af6761911467 skipped: already exists
Copying blob df9b9388f04a skipped: already exists
Copying blob 7ca62b038e32 skipped: already exists
Copying blob fb842226b602 skipped: already exists
Copying blob f574972f1e06 skipped: already exists
Copying blob d187eab9332a skipped: already exists
Copying blob 30f5083aee51 skipped: already exists
Copying blob f9d05e222640 skipped: already exists
Copying blob 6da6b0cc3f77 skipped: already exists
Copying blob 5491657e03bb skipped: already exists
Copying blob a1043afc633d done
Copying blob cdba2656a26e done
Copying blob e738b756a9b5 done
Copying blob 6984f9f643bf done
Copying blob 0882848c0ae2 done
Copying config 93ead6e9e2 done
Writing manifest to image destination
Storing signatures
time="2022-06-23T03:35:59Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8
Copying blob sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8
Copying config sha256:e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294
Writing manifest to image destination
Storing signatures
Error: runc: runc create failed: unable to start container process: exec: "echo": executable file not found in $PATH: OCI runtime attempted to invoke a command that was not found

On the minimal container:
docker run --privileged -it -u podman:podman mgoltzsche/podman:minimal ash

/ $ podman version
Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.18.3
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

/ $ cat /etc/subuid
podman:100000:65536

/ $ cat /etc/subgid
podman:100000:65536

/ $ whoami
podman

Tried to launch a container again within the minimal container, same issue. The image was pulled but the container can not be run. It might be some configuration problem with the container runtime crun in the minimal image.

/ $ docker run -it alpine:latest ash
WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers
✔ docker.io/library/alpine:latest
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob 2408cc74d12b done
Copying config e66264b987 done
Writing manifest to image destination
Storing signatures
Error: crun: executable file `ash` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found
/ $ exit
``

[awidjaja]

What I have noticed:

Edited: Please ignore this comment, it seems that I have mixed up podman info from the host instead of taking it from the minimal container.

on the host:

which runc
/usr/local/bin/runc

which crun

on the minimal container

/ $ which runc

/ $ which crun
/usr/local/bin/crun

The runtime configuration on the minimal container

  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/local/bin/runc
    version: |-
      runc version 1.1.1
      commit: v1.1.1-0-g52de29d
      spec: 1.0.2-dev
      go: go1.16.15
      libseccomp: 2.5.1

So, there is clearly a mismatch between the installed runtime crun and the configured runc.
The question is, should the configuration be changed to crun or should runc be installed in the minimal image?

[mgoltzsche]

The minimal image comes with crun instead of runc and is configured for low isolation for a lower footprint to avoid problems when trying to run podman as unprivileged user in restricted environments. Therefore seeing crun instead of runc in the minimal container is perfectly fine.
However it would indeed be odd if podman info within the minimal container indicates that runc is used although it uses crun but I cannot reproduce that - did you mix up the output maybe accidentally?

Though I was able to reproduce the error (or a similarly looking one at least) by trying to run the podman image (not the minimal one though!) as root on a raspbian arm64:

$ sudo podman run --privileged -u podman:podman mgoltzsche/podman:4.1.1 docker run alpine:latest echo hello
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for memory: mkdir /sys/fs/cgroup/memory: read-only file system 
time="2022-06-23T21:44:58Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob sha256:b3c136eddcbf2003d3180787cef00f39d46b9fd9e4623178282ad6a8d63ad3b0
Copying blob sha256:b3c136eddcbf2003d3180787cef00f39d46b9fd9e4623178282ad6a8d63ad3b0
Copying config sha256:6e30ab57aeeef1ebca8ac5a6ea05b5dd39d54990be94e7be18bb969a02d10a3f
Writing manifest to image destination
Storing signatures
Error: runc: runc create failed: unable to start container process: exec: "echo": executable file not found in $PATH: OCI runtime attempted to invoke a command that was not found

(Btw apparently the crun binary within the minimal image for arm64 is broken.)

Unfortunately on the Raspberry Pi I get the same error even as unprivileged user although on my Ubuntu desktop podman works perfectly fine as unprivileged user.

Could you please try to run podman as unprivileged user? Maybe it works for you on WSL2 as a workaround for now at least?

Either way would you mind creating an issue within the upstream podman repository?

I'll have a look if older podman versions have the same problem or rather when it appeared...

[awidjaja]

Podman info on the podman host (root user)

  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/local/bin/runc
    version: |-
      runc version 1.1.3
      commit: v1.1.3-0-g6724737
      spec: 1.0.2-dev
      go: go1.18.3
      libseccomp: 2.5.2

Podman info on the minimal image (podman user). It seems as you said, I might have mixed up with the host container, sorry about that.

  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/local/bin/crun
    version: |-
      crun version 1.4.5
      commit: c381048530aa750495cf502ddb7181f2ded5b400
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

# Full podman info on the minimal container:

podman info
WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers
host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/local/lib/podman/conmon
    version: 'conmon version 2.1.2, commit: 2bc95ee697e87d5f7b77063cf83fc32739addafe'
  cpuUtilization:
    idlePercent: 98.73
    systemPercent: 0.55
    userPercent: 0.72
  cpus: 4
  distribution:
    distribution: alpine
    version: 3.15.4
  eventLogger: file
  hostname: 21aaf3a797f9
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.102.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 71782400
  memTotal: 1505017856
  networkBackend: cni
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/local/bin/crun
    version: |-
      crun version 1.4.5
      commit: c381048530aa750495cf502ddb7181f2ded5b400
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 484544512
  swapTotal: 524288000
  uptime: 23h 54m 28.82s (Approximately 0.96 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  configFile: /podman/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.ignore_chown_errors: "true"
  graphRoot: /podman/.local/share/containers/storage
  graphRootAllocated: 269490393088
  graphRootUsed: 345808896
  graphStatus:
    Backing Filesystem: <unknown>
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /tmp/podman-run-1000/containers
  volumePath: /podman/.local/share/containers/storage/volumes
version:
  APIVersion: 4.1.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.3
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.1

As per Podman's documentation the host container can only run in priviledge mode.

Run minimal without priviledged failed

podman run -u podman:podman mgoltzsche/podman docker run alpine:latest echo hello from nested container

time="2022-06-23T23:13:56Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
Resolving "alpine" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8
Copying blob sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8
Copying config sha256:e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294
Writing manifest to image destination
Storing signatures
time="2022-06-23T23:14:08Z" level=warning msg="failed to set net.ipv6.conf.default.accept_dad sysctl: open /proc/sys/net/ipv6/conf/default/accept_dad: read-only file system"
Error: /usr/local/bin/slirp4netns failed: "WARNING: Support for seccomp is experimental\nopen(\"/dev/net/tun\"): No such file or directory\nWARNING: Support for IPv6 is experimental\nchild failed(1)\nWARNING: Support for IPv6 is experimental\n"

Run unpriviledge podman host (podman user) and tried again

whoami
podman

podman run -u podman:podman mgoltzsche/podman docker run alpine:latest echo hello from nested container

WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers
✔ docker.io/mgoltzsche/podman:latest
Trying to pull docker.io/mgoltzsche/podman:latest...
Getting image source signatures
Copying blob d187eab9332a done
Copying blob af6761911467 done
Copying blob 7ca62b038e32 done
Copying blob fb842226b602 done
Copying blob df9b9388f04a done
Copying blob f574972f1e06 done
Copying blob f9d05e222640 done
Copying blob 6da6b0cc3f77 done
Copying blob 30f5083aee51 done
Copying blob 5491657e03bb done
Copying blob cdba2656a26e done
Copying blob e738b756a9b5 done
Copying blob a1043afc633d done
Copying blob 6984f9f643bf done
Copying blob 0882848c0ae2 done
Copying config 93ead6e9e2 done
Writing manifest to image destination
Storing signatures
time="2022-06-23T23:22:28Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
time="2022-06-23T23:22:28Z" level=error msg="running `/usr/bin/newuidmap 15 0 1000 1 1 100000 65536`: newuidmap: write to uid_map failed: Operation not permitted\n"
Error: cannot setup namespace using "/usr/bin/newuidmap": exit status 1

With priviledge flag

podman run --privileged -u podman:podman mgoltzsche/podman:minimal docker run alpine:latest echo hello from nested container
Resolved "mgoltzsche/podman" as an alias (/podman/.cache/containers/short-name-aliases.conf)
Trying to pull docker.io/mgoltzsche/podman:minimal...
Getting image source signatures
Copying blob af6761911467 skipped: already exists
Copying blob f574972f1e06 skipped: already exists
Copying blob d187eab9332a skipped: already exists
Copying blob 7ca62b038e32 skipped: already exists
Copying blob fb842226b602 skipped: already exists
Copying blob f9d05e222640 skipped: already exists
Copying blob 5491657e03bb skipped: already exists
Copying blob df9b9388f04a skipped: already exists
Copying blob 30f5083aee51 skipped: already exists
Copying blob 6da6b0cc3f77 skipped: already exists
Copying blob 99389a6ec7b1 done
Copying blob 5c5b7460be86 done
Copying config 9b621f9d1e done
Writing manifest to image destination
Storing signatures
time="2022-06-23T23:25:03Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
time="2022-06-23T23:25:03Z" level=error msg="running `/usr/bin/newuidmap 15 0 1000 1 1 100000 65536`: newuidmap: write to uid_map failed: Operation not permitted\n"
Error: cannot setup namespace using "/usr/bin/newuidmap": exit status 1

[mgoltzsche]

@awidjaja oh, in that case the problem is probably caused by a misconfiguration within the alpine-baed image indeed. I must admit that I am using podman as unprivileged user only currently which is why I didn't notice the issue (and the container in container use case is an advanced one that is not covered by the e2e tests unfortunately). However the next days/weeks I won't find the time to fix it. Please feel free to create a PR in case you find a fix.

Error: cannot setup namespace using "/usr/bin/newuidmap"

This looks like you didn't install uidmap on the host as described here. Once you've installed that dependency, you should be able to use podman as unprivileged user. Please let me know if that's not the case.
UPDATE: Oh, wait, this is happening within the podman image and therefore it shouldn't be related to the host.

[mgoltzsche]

@awidjaja the uidmap problem was fixed with the 4.4.1 release (the config shipped by this repo was wrong indeed 😅). Feel free to reopen the issue if that's not the case.