svdvoort / radiomics-gpu-server Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
How to test: Launch a multi-processed task (I only tested with python multi-processing) from either the command line directly or by submitting a SLURM job.
Current result: The SLURM job will only use one core, contrary to the command line one which will use as many cores as it needs (and can) for its processes.
Expected: both should behave identically regarding multiprocessing, using as many cores as needed
Node is set to the 'drained' state with reason 'batch job complete failure'.
Log:
[2018-08-09T20:32:45.578] sched: Allocate JobID=47 NodeList=bigr-nzxt-7 #CPUs=12 Partition=GPU
[2018-08-10T10:05:13.325] _slurm_rpc_kill_job: REQUEST_KILL_JOB job 47 uid 1008
[2018-08-10T10:06:14.007] error: slurmd error running JobId=47 on node(s)=bigr-nzxt-7: Kill task failed
[2018-08-10T10:06:14.007] drain_nodes: node bigr-nzxt-7 state set to DRAIN
[2018-08-10T10:06:14.377] cleanup_completing: job 47 completion process took 61 seconds
Apparently, the job cleanup took too long. If this happens again we might want to increase the UnkillableStepTimeout.
The unloading of the other module should be done automatically, user interaction should not be required.
module purge
module load compilers/python/3.6.6
module load tensorflow/1.9.0
python3
>>>import numpy as np
outputs the following error:
Traceback (most recent call last):
File "/packages/tensorflow/1.9.0/Python-3.5.6/numpy/core/init.py", line 16, in
from . import multiarray
ImportError: cannot import name 'multiarray'During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "/packages/tensorflow/1.9.0/Python-3.5.6/numpy/init.py", line 142, in
from . import add_newdocs
File "/packages/tensorflow/1.9.0/Python-3.5.6/numpy/add_newdocs.py", line 13, in
from numpy.lib import add_newdoc
File "/packages/tensorflow/1.9.0/Python-3.5.6/numpy/lib/init.py", line 8, in
from .type_check import *
File "/packages/tensorflow/1.9.0/Python-3.5.6/numpy/lib/type_check.py", line 11, in
import numpy.core.numeric as _nx
File "/packages/tensorflow/1.9.0/Python-3.5.6/numpy/core/init.py", line 26, in
raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, trygit clean -xdf
(removes all
files not under version control). Otherwise reinstall numpy.Original error was: cannot import name 'multiarray'
If Python 3.5.6 or TensorFlow 1.10.0 is loaded instead, it works
After doing module load python/3.X
the default python
command will still default to the pre-installed python in the system. It would be more logical to have the default python
command default to python3 in this case.
When the system is rebooted (this seems to be the case, although need to check if this is consistent), when trying to run nvidia-smi
, it returns the error: 'NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.'
running nvidia-settings
returns the error:
ERROR: NVIDIA driver is not loaded
ERROR: Unable to load info from any available system
Although the driver is still installed and unchanged.
This issue is solved by running the cuda install script again, which will reinstall the driver. However, of course the driver is already installed so this should not be needed every time.
Graphviz (https://www.graphviz.org/) is a useful library to make graphs.
It's used as the backend of some tools, such as https://keras.io/visualization/, to visualize the computational graph of neural networks. However python interfaces for it don't include the binaries which must be installed separately with root access (to use apt).
Certain versions of python (for example 2.7.12) do not compile with ssl support. It will give the error:
Failed to build these modules:
_hashlib _ssl
As a result pip cannot be installed and used because pip uses https and thus requires ssl support.
The problem seems to lie in the fact that some versions of python do not support openssl1.1, but rely on openssl1.0.
It has been suggested to add the openssl library to the path, like here. This has been implemented in a9ba37f, however this did not solve the issue. Even when using a custom install of the openssl1.0 library. It is suggested that changes be made to the python installs scripts, but this seems difficult to automate.
When trying to load matplotlib, the following error occurs:
File "/home/fcalvet/clm_seg/basicFunctions.py", line 1, in
import matplotlib
File "/home/fcalvet/CNN4/lib/python3.6/site-packages/matplotlib/init.py", line 127, in
from . import cbook
File "/home/fcalvet/CNN4/lib/python3.6/site-packages/matplotlib/cbook/init.py", line 13, in
import bz2
File "/packages/python/3.6.6/lib/python3.6/bz2.py", line 23, in
from _bz2 import BZ2Compressor, BZ2Decompressor
ModuleNotFoundError: No module named '_bz2'
This is probably because bzip2 wasn't available during the python compilation, according to https://stackoverflow.com/questions/8115280/importerror-no-module-named-bz2-for-python-2-7-2#14123902. A quick way to check is python(3) -c "import bz2; print(bz2.__doc__)"
However this should be solved by a9ba37f, so may be re-compiling the Pythons is needed
PS: thanks for adding the conflict error when trying to load multiples Python versions
I have tried to create a virtual environment to run python 2.7.15 but the package virtualenv is not available
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.