Multiple CPUs? about deep_complex_networks

[obilaniu]

@austinmw

A runtime on the order of 4 hours is consistent with running on CPU. With gpuarray 0.7.5, a P100 GPU and THEANO_FLAGS="mode=FAST_RUN,device=cuda,floatX=float32,gpuarray.preallocate=1", Musicnet epochs take on the order of 100-150 seconds each.

I did not code the Musicnet aspect, so I can't vouch for its resumability.

[obilaniu]

@austinmw Approximately the setup for Musicnet:
Theano 1.0.1
libgpuarray/pygpu 0.7.5 (they're from the same repo)
cuDNN 6.0.21
CUDA Toolkit 8.0 (I believe).

Those are the newest Theano/libgpuarray releases and somewhat old cuDNN/CUDA libraries. Still works very well.

[obilaniu]

The code will run on multiple CPUs provided you've linked it with the multithreaded builds of BLAS libraries like MKL or OpenBLAS, and have not undertaken steps to disable multithreading (such as through the setting of certain OpenMP environment variables).

There is no support at present for multiple GPUs, and we do not foresee adding it to the current Keras+Theano code because Theano itself is being sunset.

A future rewrite of this codebase to another framework may possibly support multiple GPUs.

[austinmw]

I'm trying out the demo and started training. It took about 4 hours to complete 1 epoch. Can I stop/restart training and test at any time? By default it's set to run for 200 epochs which for my system would be ~1 month straight. Did you find this amount of training is needed for good performance on musicnet?

[austinmw]

Thanks for the suggestion. Do you use latest theano/libgpuarray/pygpu releases?

[austinmw]

Thanks, had to edit the .theanorc with cuda path, but got it working with CUDA 9.0!