Error when running docker about isaac_ros_common

[hemalshahNV]

The run_dev.sh script should not be run with sudo. The script sets up an admin user in the container using your user id and group which does nothing work when running as root under sudo. We’ll add a check in the script to prevent this and provide guidance. Your user needs to be in the docker group to not require sudo when running docker client commands. Run ‘ sudo usermod -aG docker $USER’ from here: https://docs.docker.com/engine/install/linux-postinstall/ to update that.

[kaganerunsal]

Hello. thank you very much for the feedback. I did what you suggested, basically:

1. Removed the containersdocker rm -vf $(docker ps -a -q)
2. Removed the images docker rmi -f $(docker images -a -q)
3. Added group sudo groupadd docker
4. Added user to docker group sudo usermod -aG docker $USER
5. Run script without sudo ./run_dev.sh to build it

I have attached the output of the build and at the end, you can see the same error I have encountered.

output.txt

I am still suspecting it might be related with nvidia-container libraries. But all my libraries are up to date. Do you have any other suggestion? Thank you in advance.

Note: It builds successfully but have an error when it runs docker.

[hemalshahNV]

This error seems very different, you are right. Did you happen to run out of memory?

…

________________________________ From: Liquidesigner ***@***.***> Sent: Monday, November 1, 2021 10:09:29 AM To: NVIDIA-ISAAC-ROS/isaac_ros_common ***@***.***> Cc: Hemal Shah ***@***.***>; Comment ***@***.***> Subject: Re: [NVIDIA-ISAAC-ROS/isaac_ros_common] Error when running docker (Issue #1) Hello. thank you very much for the feedback. I did what you suggested, basically: 1. Removed the containersdocker rm -vf $(docker ps -a -q) 2. Removed the images docker rmi -f $(docker images -a -q) 3. Added group sudo groupadd docker 4. Added user to docker group sudo usermod -aG docker $USER 5. Run script without sudo ./run_dev.sh to build it I have attached the output of the build and at the end, you can see the same error I have encountered. output.txt<https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common/files/7454970/output.txt> I am still suspecting it might be related with nvidia-container libraries. But all my libraries are up to date. Do you have any other suggestion? Thank you in advance. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#1 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ASS44RWXEQQC6R32ZDLHZ6DUJ3CUTANCNFSM5G7HBTEA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[kaganerunsal]

I think I have enough memory and drive. Here is my jtop output while running building docke

r:

[hemalshahNV]

The log spew after "Running isaac_ros_dev-aarch64-container" indicates that the nvidia container runtime could have run out of swap memory perhaps which is set to ~4GB max. This Xavier NX is running JP 4.6, so nothing unusual there and we've tested on Xavier NX and have not run into this yet. Anything else you can tell us about your setup you think could be relevant, including anything after flashing to JetPack 4.6?

Running isaac_ros_dev-aarch64-container
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: src: /usr/lib/aarch64-linux-gnu/libcudnn.so.8, src_lnk: libcudnn.so.8.2.1, ... , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig.real --device=all --compute --compat32 --graphics --utility --video --display --pid=13679 /var/lib/docker/overlay2/3809d20985900c3386f4edc47c970eea0d66370dcd769db4871635145a20e776/merged]
nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/3809d20985900c3386f4edc47c970eea0d66370dcd769db4871635145a20e776/merged/dev/nvhost-nvdla0: cannot allocate memory: unknown.

[kaganerunsal]

I flashed Jetpack 4.6 by using SDK manager, possibly just after i flashed it, I updated the repo withsudo apt update and upgrade the packages sudo apt-get upgrade. However since default repos exist on apt, upgrade should not create a problem.

[FPSychotic]

i get this error, relevant info is Im in 20.04,Xavier NX, opencv 4.4 with cuda.

please , is it possible install this without docker?

error:
Sending build context to Docker daemon 22.02kB Step 1/38 : ARG BASE_IMAGE="dustynv/ros:foxy-ros-base-l4t-r32.6.1" Step 2/38 : FROM ${BASE_IMAGE} foxy-ros-base-l4t-r32.6.1: Pulling from dustynv/ros b9bb7af7248f: Pulling fs layer 334a570c08f5: Pulling fs layer c8bd4a89e71c: Pulling fs layer 0a8ce4f08307: Pulling fs layer 2e163bfa50fc: Pulling fs layer 8d6d8c86a148: Pull complete b483bad8509f: Pull complete 8da3c6b29858: Pull complete a3e94453e320: Pull complete 40ad1f92affd: Pull complete 8ac6e2f9e2a0: Pull complete cc932e62bbab: Pull complete d77841654f58: Pull complete 2bb42f0b6424: Pull complete 954e18eab77a: Pull complete facc6f8bcc9b: Pull complete d4b32f371445: Pull complete 8abe66ee445c: Pull complete 2bc2b7a54d18: Pull complete c1a957de25c3: Pull complete 4811d7ce2890: Pull complete 7c40556846ce: Pull complete e00c9dd19f7d: Pull complete d4ab5f1a7659: Pull complete caa4bb4a1a92: Pull complete 92aa50f1eff2: Pull complete 44c4917d20dd: Pull complete eb14fae7b412: Pull complete Digest: sha256:3271b330732647c65f178980b5e6437d2b9ea03627ac0104e0d78ebb79cd55bc Status: Downloaded newer image for dustynv/ros:foxy-ros-base-l4t-r32.6.1 ---> bc3b88966255 Step 3/38 : ENV DEBIAN_FRONTEND=noninteractive ---> Running in c5208e12b3c1 Removing intermediate container c5208e12b3c1 ---> cdd86af29294 Step 4/38 : RUN apt-get update && apt-get install -y build-essential cmake curl git lsb-release sudo tar unzip vim wget software-properties-common && rm -rf /var/lib/apt/lists/* ---> Running in eacafa9d00af failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #1:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request: unknown ~/isaac_ros_common-main/scripts

[hemalshahNV]

What is it that you would like to use without Docker? Depending on the Isaac ROS package, the Docker container and the run_dev.sh script are optional. This error looks like the Nvidia Container Toolkit is not setup properly (configured for you in JP 4.6). We have only qualified Isaac ROS (packages and Docker containers) on Jetpack 4.6 (based on Ubuntu 18.04), so there will be issues on Ubuntu 20.04.

[FPSychotic]

What is it that you would like to use without Docker? Depending on the Isaac ROS package, the Docker container and the run_dev.sh script are optional. This error looks like the Nvidia Container Toolkit is not setup properly (configured for you in JP 4.6). We have only qualified Isaac ROS (packages and Docker containers) on Jetpack 4.6 (based on Ubuntu 18.04), so there will be issues on Ubuntu 20.04.

I would like can control what I use, docker do not fit to my application, and it is a limitating thing. The good of linux is can compile and apt, why force to use containers? In fact it is not working, and now I need depend of nvida, ubuntu, the dependencies maintainers and now of docker and the use that nvidia want ro do of it.
Thanks, I'm already suffering L4T, please don't make me suffer docker too, and mainly because don't have sense.
Please add the way and guide to compile from source everything, it is what everyone does

[hemalshahNV]

I would like can control what I use, docker do not fit to my application, and it is a limitating thing. The good of linux is can compile and apt, why force to use containers? In fact it is not working, and now I need depend of nvida, ubuntu, the dependencies maintainers and now of docker and the use that nvidia want ro do of it. Thanks, I'm already suffering L4T, please don't make me suffer docker too, and mainly because don't have sense. Please add the way and guide to compile from source everything, it is what everyone does

I understand Docker does not fit your application. We use a Docker-based development environment but we had to also adopt a Docker container for runtime as well to make it as plug-and-play as possible for our users. Docker containers let us assume more about the environment you're running on so we can qualify our packages more effectively. Triton is very sensitive to the versions of TensorRT, cuDNN, etc. and on x86 where there is no equivalent of Jetpack, we could not reasonably expect our users to solve that 3-SAT problem easily. On aarch64, we have the right versions of everything but we wanted to helps users avoid compiling ROS2 from source because precompiled binaries are not available. With the Nvidia Container Toolkit, Docker has minimal overhead and has many advantages for production robots (complete builds, easy deployment, fast upgrades).

Of course, if you want to run without the Docker containers, you can try. On Jetpack, it should be easy to apply to the host machine what we have in Dockerfile.aarch64.base, but x86 may be more difficult to get right. We'll look for ways to help re-enable users running "bare metal," but for now, this was the best solution so users could run without tricky setup.

[FPSychotic]

Thanks by your kind and reasonable explanation. I fixed my problem being able to get what I need, thanks by your time and effort.
You al did .a hard job in this, I recognise that and appreciate it

[kaganerunsal]

Hi hemalshahNV,

I managed to successfully build the docker with a clean installation of Jetpack 4.6 image again. With my Jetson Xavier NX, I am using e-CAM50_CUNX camera from e-con Systems. I have discovered that the drivers of this camera is compatible with Jetpack 4.5 and after I installed the binaries, i could not run the docker built. Possibly there are some downgrades of nvidia container libraries. I asked them to send me Jetpack 4.6 version of camera drivers. Currently, i am using another camera, arducam IMX219, which is natively supported, no need to install any drivers.