I have attempted to get this running, but have run into an import error (librccl.so)
When I search for that file I find a few. Has anyone dealt with this issue before?
(sagemaker_venv) raphy@raphy:~/Projects/deepracer/rl_coach$ sudo find / -name librccl.so -exec file {} \; -exec printf "\n" \;
find: ‘/run/user/1000/gvfs’: Permission denied
/opt/rocm/rccl/lib/librccl.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
/opt/rocm/lib/librccl.so: symbolic link to ../rccl/lib/librccl.so
/home/raphy/Projects/virtual_envs/tf_rocm/lib/python3.7/site-packages/tensorflow/include/external/local_config_rocm/rocm/rocm/lib/librccl.so: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, not stripped
/var/lib/docker/overlay2/b64506e1a7e7ffdcb7dd183e73cd5d79fcb5f5e3ce847c2103180f2cdd13bd5c/diff/usr/local/lib/python3.6/dist-packages/tensorflow/include/external/local_config_rocm/rocm/rocm/lib/librccl.so: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, not stripped
(sagemaker_venv) raphy@raphy:~/Projects/deepracer/rl_coach$ python rl`_deepracer_coach_robomaker.py
Looking for config file: /home/raphy/.sagemaker/config.yaml
Model checkpoints and other metadata will be stored at: s3://bucket/rl-deepracer-sagemaker
Uploading to s3://bucket/rl-deepracer-sagemaker
WARNING:sagemaker:Parameter `image_name` is specified, `toolkit`, `toolkit_version`, `framework` are going to be ignored when choosing the image.
s3.ServiceResource()
Using provided s3_client
INFO:sagemaker:Creating training-job with name: rl-deepracer-sagemaker
Starting training job
Using /home/raphy/Projects/deepracer/containers for container temp files
Using /home/raphy/Projects/deepracer/containers for container temp files
Trying to launch image: crr0004/sagemaker-rl-tensorflow:amd
Creating tmp9whzr7mk_algo-1-g8xmx_1 ... done
Attaching to tmp9whzr7mk_algo-1-g8xmx_1
algo-1-g8xmx_1 | $1 is train
algo-1-g8xmx_1 | In train start.sh
algo-1-g8xmx_1 | Current host is "algo-1-g8xmx"
algo-1-g8xmx_1 | Compiling changehostname.c
algo-1-g8xmx_1 | Done Compiling changehostname.c
algo-1-g8xmx_1 | 21:C 07 Aug 2019 17:22:17.267 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
algo-1-g8xmx_1 | 21:C 07 Aug 2019 17:22:17.267 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=21, just started
algo-1-g8xmx_1 | 21:C 07 Aug 2019 17:22:17.267 # Configuration loaded
algo-1-g8xmx_1 | _._
algo-1-g8xmx_1 | _.-``__ ''-._
algo-1-g8xmx_1 | _.-`` `. `_. ''-._ Redis 5.0.5 (00000000/0) 64 bit
algo-1-g8xmx_1 | .-`` .-```. ```\/ _.,_ ''-._
algo-1-g8xmx_1 | ( ' , .-` | `, ) Running in standalone mode
algo-1-g8xmx_1 | |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
algo-1-g8xmx_1 | | `-._ `._ / _.-' | PID: 21
algo-1-g8xmx_1 | `-._ `-._ `-./ _.-' _.-'
algo-1-g8xmx_1 | |`-._`-._ `-.__.-' _.-'_.-'|
algo-1-g8xmx_1 | | `-._`-._ _.-'_.-' | http://redis.io
algo-1-g8xmx_1 | `-._ `-._`-.__.-'_.-' _.-'
algo-1-g8xmx_1 | |`-._`-._ `-.__.-' _.-'_.-'|
algo-1-g8xmx_1 | | `-._`-._ _.-'_.-' |
algo-1-g8xmx_1 | `-._ `-._`-.__.-'_.-' _.-'
algo-1-g8xmx_1 | `-._ `-.__.-' _.-'
algo-1-g8xmx_1 | `-._ _.-'
algo-1-g8xmx_1 | `-.__.-'
algo-1-g8xmx_1 |
algo-1-g8xmx_1 | 21:M 07 Aug 2019 17:22:17.268 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
algo-1-g8xmx_1 | 21:M 07 Aug 2019 17:22:17.268 # Server initialized
algo-1-g8xmx_1 | 21:M 07 Aug 2019 17:22:17.268 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
algo-1-g8xmx_1 | 21:M 07 Aug 2019 17:22:17.268 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
algo-1-g8xmx_1 | 21:M 07 Aug 2019 17:22:17.268 * Ready to accept connections
algo-1-g8xmx_1 | 07/08/2019 17:22:17 passing arg to libvncserver: -rfbport
algo-1-g8xmx_1 | 07/08/2019 17:22:17 passing arg to libvncserver: 5800
algo-1-g8xmx_1 | 07/08/2019 17:22:17 x11vnc version: 0.9.13 lastmod: 2011-08-10 pid: 22
algo-1-g8xmx_1 | 07/08/2019 17:22:17
algo-1-g8xmx_1 | 07/08/2019 17:22:17 wait_for_client: WAIT:0
algo-1-g8xmx_1 | 07/08/2019 17:22:17
algo-1-g8xmx_1 | 07/08/2019 17:22:17 initialize_screen: fb_depth/fb_bpp/fb_Bpl 24/32/2560
algo-1-g8xmx_1 | 07/08/2019 17:22:17
algo-1-g8xmx_1 | 07/08/2019 17:22:17 Listening for VNC connections on TCP port 5800
algo-1-g8xmx_1 | 07/08/2019 17:22:17 Listening for VNC connections on TCP6 port 5900
algo-1-g8xmx_1 | 07/08/2019 17:22:17 Listening also on IPv6 port 5800 (socket 6)
algo-1-g8xmx_1 | 07/08/2019 17:22:17
algo-1-g8xmx_1 |
algo-1-g8xmx_1 | The VNC desktop is: ceca255b3de9:5800
algo-1-g8xmx_1 | 07/08/2019 17:22:17 possible alias: ceca255b3de9::5800
algo-1-g8xmx_1 | PORT=5800
algo-1-g8xmx_1 | Reporting training FAILURE
algo-1-g8xmx_1 | framework error:
algo-1-g8xmx_1 | Traceback (most recent call last):
algo-1-g8xmx_1 | File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
algo-1-g8xmx_1 | from tensorflow.python.pywrap_tensorflow_internal import *
algo-1-g8xmx_1 | File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
algo-1-g8xmx_1 | _pywrap_tensorflow_internal = swig_import_helper()
algo-1-g8xmx_1 | File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
algo-1-g8xmx_1 | _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
algo-1-g8xmx_1 | File "/usr/lib/python3.6/imp.py", line 243, in load_module
algo-1-g8xmx_1 | return load_dynamic(name, filename, file)
algo-1-g8xmx_1 | File "/usr/lib/python3.6/imp.py", line 343, in load_dynamic
algo-1-g8xmx_1 | return _load(spec)
algo-1-g8xmx_1 | ImportError: librccl.so: cannot open shared object file: No such file or directory
algo-1-g8xmx_1 |
algo-1-g8xmx_1 | During handling of the above exception, another exception occurred:
algo-1-g8xmx_1 |
algo-1-g8xmx_1 | Traceback (most recent call last):
algo-1-g8xmx_1 | File "/usr/local/lib/python3.6/dist-packages/sagemaker_containers/_trainer.py", line 60, in train
algo-1-g8xmx_1 | framework = importlib.import_module(framework_name)
algo-1-g8xmx_1 | File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
algo-1-g8xmx_1 | return _bootstrap._gcd_import(name[level:], package, level)
algo-1-g8xmx_1 | File "<frozen importlib._bootstrap>", line 994, in _gcd_import
algo-1-g8xmx_1 | File "<frozen importlib._bootstrap>", line 971, in _find_and_load
algo-1-g8xmx_1 | File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
algo-1-g8xmx_1 | File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
algo-1-g8xmx_1 | File "<frozen importlib._bootstrap_external>", line 678, in exec_module
algo-1-g8xmx_1 | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
algo-1-g8xmx_1 | File "/usr/local/lib/python3.6/dist-packages/sagemaker_tensorflow_container/training.py", line 24, in <module>
algo-1-g8xmx_1 | import tensorflow as tf
algo-1-g8xmx_1 | File "/usr/local/lib/python3.6/dist-packages/tensorflow/__init__.py", line 28, in <module>
algo-1-g8xmx_1 | from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
algo-1-g8xmx_1 | File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/__init__.py", line 49, in <module>
algo-1-g8xmx_1 | from tensorflow.python import pywrap_tensorflow
algo-1-g8xmx_1 | File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>
algo-1-g8xmx_1 | raise ImportError(msg)
algo-1-g8xmx_1 | ImportError: Traceback (most recent call last):
algo-1-g8xmx_1 | File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
algo-1-g8xmx_1 | from tensorflow.python.pywrap_tensorflow_internal import *
algo-1-g8xmx_1 | File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
algo-1-g8xmx_1 | _pywrap_tensorflow_internal = swig_import_helper()
algo-1-g8xmx_1 | File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
algo-1-g8xmx_1 | _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
algo-1-g8xmx_1 | File "/usr/lib/python3.6/imp.py", line 243, in load_module
algo-1-g8xmx_1 | return load_dynamic(name, filename, file)
algo-1-g8xmx_1 | File "/usr/lib/python3.6/imp.py", line 343, in load_dynamic
algo-1-g8xmx_1 | return _load(spec)
algo-1-g8xmx_1 | ImportError: librccl.so: cannot open shared object file: No such file or directory
algo-1-g8xmx_1 |
algo-1-g8xmx_1 |
algo-1-g8xmx_1 | Failed to load the native TensorFlow runtime.