svdvoort / radiomics-gpu-server Goto Github PK

How to test: Launch a multi-processed task (I only tested with python multi-processing) from either the command line directly or by submitting a SLURM job.
Current result: The SLURM job will only use one core, contrary to the command line one which will use as many cores as it needs (and can) for its processes.
Expected: both should behave identically regarding multiprocessing, using as many cores as needed

Node is set to the 'drained' state with reason 'batch job complete failure'.

Log:
[2018-08-09T20:32:45.578] sched: Allocate JobID=47 NodeList=bigr-nzxt-7 #CPUs=12 Partition=GPU
[2018-08-10T10:05:13.325] _slurm_rpc_kill_job: REQUEST_KILL_JOB job 47 uid 1008
[2018-08-10T10:06:14.007] error: slurmd error running JobId=47 on node(s)=bigr-nzxt-7: Kill task failed
[2018-08-10T10:06:14.007] drain_nodes: node bigr-nzxt-7 state set to DRAIN
[2018-08-10T10:06:14.377] cleanup_completing: job 47 completion process took 61 seconds

Apparently, the job cleanup took too long. If this happens again we might want to increase the UnkillableStepTimeout.

The unloading of the other module should be done automatically, user interaction should not be required.

module purge
module load compilers/python/3.6.6
module load tensorflow/1.9.0
python3
>>>import numpy as np

outputs the following error:

Traceback (most recent call last):
File "/packages/tensorflow/1.9.0/Python-3.5.6/numpy/core/init.py", line 16, in
from . import multiarray
ImportError: cannot import name 'multiarray'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/packages/tensorflow/1.9.0/Python-3.5.6/numpy/init.py", line 142, in
from . import add_newdocs
File "/packages/tensorflow/1.9.0/Python-3.5.6/numpy/add_newdocs.py", line 13, in
from numpy.lib import add_newdoc
File "/packages/tensorflow/1.9.0/Python-3.5.6/numpy/lib/init.py", line 8, in
from .type_check import *
File "/packages/tensorflow/1.9.0/Python-3.5.6/numpy/lib/type_check.py", line 11, in
import numpy.core.numeric as _nx
File "/packages/tensorflow/1.9.0/Python-3.5.6/numpy/core/init.py", line 26, in
raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try git clean -xdf (removes all
files not under version control). Otherwise reinstall numpy.

Original error was: cannot import name 'multiarray'

If Python 3.5.6 or TensorFlow 1.10.0 is loaded instead, it works

After doing module load python/3.X the default python command will still default to the pre-installed python in the system. It would be more logical to have the default pythoncommand default to python3 in this case.

When the system is rebooted (this seems to be the case, although need to check if this is consistent), when trying to run nvidia-smi, it returns the error: 'NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.'

running nvidia-settings returns the error:

ERROR: NVIDIA driver is not loaded
ERROR: Unable to load info from any available system

Although the driver is still installed and unchanged.

This issue is solved by running the cuda install script again, which will reinstall the driver. However, of course the driver is already installed so this should not be needed every time.

Graphviz (https://www.graphviz.org/) is a useful library to make graphs.
It's used as the backend of some tools, such as https://keras.io/visualization/, to visualize the computational graph of neural networks. However python interfaces for it don't include the binaries which must be installed separately with root access (to use apt).

Certain versions of python (for example 2.7.12) do not compile with ssl support. It will give the error:

Failed to build these modules:
_hashlib           _ssl

As a result pip cannot be installed and used because pip uses https and thus requires ssl support.
The problem seems to lie in the fact that some versions of python do not support openssl1.1, but rely on openssl1.0.

It has been suggested to add the openssl library to the path, like here. This has been implemented in a9ba37f, however this did not solve the issue. Even when using a custom install of the openssl1.0 library. It is suggested that changes be made to the python installs scripts, but this seems difficult to automate.

When trying to load matplotlib, the following error occurs:

File "/home/fcalvet/clm_seg/basicFunctions.py", line 1, in
import matplotlib
File "/home/fcalvet/CNN4/lib/python3.6/site-packages/matplotlib/init.py", line 127, in
from . import cbook
File "/home/fcalvet/CNN4/lib/python3.6/site-packages/matplotlib/cbook/init.py", line 13, in
import bz2
File "/packages/python/3.6.6/lib/python3.6/bz2.py", line 23, in
from _bz2 import BZ2Compressor, BZ2Decompressor
ModuleNotFoundError: No module named '_bz2'

This is probably because bzip2 wasn't available during the python compilation, according to https://stackoverflow.com/questions/8115280/importerror-no-module-named-bz2-for-python-2-7-2#14123902. A quick way to check is python(3) -c "import bz2; print(bz2.__doc__)"

However this should be solved by a9ba37f, so may be re-compiling the Pythons is needed

PS: thanks for adding the conflict error when trying to load multiples Python versions

I have tried to create a virtual environment to run python 2.7.15 but the package virtualenv is not available