[Help] Can't start created container: executable file not found in $PATH about python-podman

[stevenstetzler]

I think I found a hint toward the issue by doing a diff on the container inspect outputs. It looks like the PATH environment variable gets messed up in container creation:

For the Python API created container (incorrect PATH):

            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin",
                "TERM=xterm",
                "HOSTNAME=a07d93be42c8",
                "LANGUAGE=en_US.UTF-8",
                "CONDA_VERSION=4.8.3",
                "NB_USER=jovyan",
                "MINICONDA_VERSION=4.8.3",
                "SHELL=/bin/bash",
                "LC_ALL=en_US.UTF-8",
                "container=podman",
                "NB_GID=100",
                "LANG=en_US.UTF-8",
                "DEBIAN_FRONTEND=noninteractive",
                "HOME=/home/jovyan",
                "NB_UID=1000",
                "CONDA_DIR=/opt/conda",
                "MINICONDA_MD5=d63adf39f2c220950a063e0529d4ff74"
            ],

and for the CLI created container (correct PATH):

            "Env": [
                "PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "CONDA_VERSION=4.8.3",
                "LC_ALL=en_US.UTF-8",
                "LANG=en_US.UTF-8",
                "CONDA_DIR=/opt/conda",
                "NB_GID=100",
                "NB_USER=jovyan",
                "DEBIAN_FRONTEND=noninteractive",
                "MINICONDA_VERSION=4.8.3",
                "container=podman",
                "LANGUAGE=en_US.UTF-8",
                "MINICONDA_MD5=d63adf39f2c220950a063e0529d4ff74",
                "NB_UID=1000",
                "SHELL=/bin/bash",
                "HOME=/home/jovyan",
                "HOSTNAME="
            ],

This is a bit confusing as it seems to me that the PATH is being taken correctly from the image before creation: https://github.com/containers/python-podman/blob/master/podman/libs/images.py#L44-L48 in the Python API, and I can see from the log output that the config sent to the varlink and go APIs appears correct: (from above)

DEBUG:root:Image f98c2844eec476cbb932cb58def069bcd5cff8f674978881fc055bcf0693ec19: create config: {'image_id': 'f98c2844eec476cbb932cb58def069bcd5cff8f674978881fc055bcf0693ec19', 'command': ['start-notebook.sh'], 'env': {'PATH': '/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'DEBIAN_FRONTEND': 'noninteractive', 'CONDA_DIR': '/opt/conda', 'SHELL': '/bin/bash', 'NB_USER': 'jovyan', 'NB_UID': '1000', 'NB_GID': '100', 'LC_ALL': 'en_US.UTF-8', 'LANG': 'en_US.UTF-8', 'LANGUAGE': 'en_US.UTF-8', 'HOME': '/home/jovyan', 'MINICONDA_VERSION': '4.8.3', 'MINICONDA_MD5': 'd63adf39f2c220950a063e0529d4ff74', 'CONDA_VERSION': '4.8.3'}, 'image': 'docker.io/jupyterhub/singleuser:latest', 'labels': {'maintainer': 'Jupyter Project <[email protected]>'}, 'net_mode': 'bridge', 'network': 'bridge', 'args': ['docker.io/jupyterhub/singleuser:latest', 'start-notebook.sh']}

I can actually fix the issue by setting the env keyword during container creation to something new, which overrides the default set by the Python API:

# up log level
import logging
logging.getLogger().setLevel(logging.DEBUG)
# action
import podman
with podman.Client() as client:
    image_id = client.images.pull("jupyterhub/singleuser:latest")
    image = client.images.get(image_id)
    container = image.create(
        env={
            "FOO": "BAR"
        }
    )
    print(container.inspect().config.get("env"))
    container.start()

produces

DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da2c44828>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da2c44828>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da264d3c8>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da264d3c8>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da25a7748>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da25a7748>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da26244e0>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da26244e0>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da263a898>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da263a898>
DEBUG:root:Image f98c2844eec476cbb932cb58def069bcd5cff8f674978881fc055bcf0693ec19: create config: {'image_id': 'f98c2844eec476cbb932cb58def069bcd5cff8f674978881fc055bcf0693ec19', 'env': {'FOO': 'BAR'}, 'command': ['start-notebook.sh'], 'image': 'docker.io/jupyterhub/singleuser:latest', 'labels': {'maintainer': 'Jupyter Project <[email protected]>'}, 'net_mode': 'bridge', 'network': 'bridge', 'args': ['docker.io/jupyterhub/singleuser:latest', 'start-notebook.sh']}
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da24f4d68>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da24f4d68>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da2455cc0>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da2455cc0>
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da2455518>
DEBUG:root:LocalClient closed varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da2455518>
['PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'TERM=xterm', 'HOSTNAME=', 'SHELL=/bin/bash', 'CONDA_VERSION=4.8.3', 'container=podman', 'DEBIAN_FRONTEND=noninteractive', 'LANGUAGE=en_US.UTF-8', 'LC_ALL=en_US.UTF-8', 'MINICONDA_VERSION=4.8.3', 'MINICONDA_MD5=d63adf39f2c220950a063e0529d4ff74', 'LANG=en_US.UTF-8', 'HOME=/home/jovyan', 'CONDA_DIR=/opt/conda', 'NB_USER=jovyan', 'NB_UID=1000', 'NB_GID=100']
DEBUG:root:LocalClient opened varlink connection <varlink.client.SimpleClientInterfaceHandler object at 0x7f5da25c97f0>
DEBUG:root:Starting Container "79912b88ed238ade23bd67d806fe7591547cb0ab26acdb3ccbaaba3da6adcdf2"
DEBUG:root:Started Container "79912b88ed238ade23bd67d806fe7591547cb0ab26acdb3ccbaaba3da6adcdf2"

and creates the container successfully. I print out the environment from the Container inspect in the output above and see that the PATH is set correctly using the image defaults. However, the FOO=BAR environment variable is not set in the container. I think the PATH gets set from the image in the go API: ParseCreateOpts (https://github.com/containers/podman/blob/v2.0.6/pkg/varlinkapi/create.go#L504-L556) called by CreateContainer (https://github.com/containers/podman/blob/v2.0.6/pkg/varlinkapi/create.go#L184).

Is this a problem of the communication between the Python API and the Varlink API?