MultiGPU Tensorflow about tensorflow-examples

[aymericdamien]

By default, Tensorflow will use all available GPUs, so to use some specific gpu, you have to use "CUDA_VISIBLE_DEVICES" flag, and specify all device ids that you want to use.
For example: export CUDA_VISIBLE_DEVICES="0,1"

[saonim]

Thanks for your response. This works!

I have one more question. My goal here is to be able to run Alexnet code on multiple GPUs.

With multi-GPU setting (CUDA_VISIBLE_DEVICE flag on), it allocates on all specified devices (GPU Memory usage is high on Nvidia-smi), but does not do computations (no memory utilization from Nvidia-smi while the program is running) on none except one. Any suggestion to make it truly parallel?

[aymericdamien]

Yes, because you have to allocate yourself operations to specific gpu, Tensorflow can't do it automatically.

with tf.device('/gpu:0'):
   ...
with tf.device('/gpu:1'):
   ...

But in order to efficiently use multiple GPU to train neural networks, you have to coordinate training. I suggest you check the official tutorial, they update parameters synchronously after each GPU finished to process a batch of data.
https://www.tensorflow.org/versions/r0.7/tutorials/deep_cnn/index.html#training-a-model-using-multiple-gpu-cards