CIFAR multi-gpu example not actually utilizing GPUs about tensorflowonspark

[HassebJ]

I was able to fix the issue. On my system echo $CUDA_VISIBLE_DEVICES=NoDevFiles for some reason.
I have two GPUs installed so setting export CUDA_VISIBLE_DEVICES=0,1 fixes the issue and TensorFlow is able to use both of them.

[HassebJ]

Environment
CUDA: 8
CUDNN: 5.0
Python: 2.7.3
Tensorflow: 1.0
Hadoop 2.7.3
Spark 2.0.2

[leewyang]

Did you install the GPU version of Tensorflow? If so, please search your yarn logs for a statement that looks like ======== worker:0 ========. You should see a following statement Using GPU: 0 or Using CPU.

[HassebJ]

As it is indicated in the GetStarted_YARN guideline, I checked out the jun_r1.0 branch as I wanted to use TensorFlow 1.0 w/ RDMA as my cluster is equipped with Infiniband.

Does this branch have GPU support?

[HassebJ]

I searched through my YARN logs and found the following:

2017-05-12 19:30:09,956 ERROR (MainThread-26814) 0: Failed to allocate GPU, trying again...
2017-05-12 19:30:30,218 INFO (MainThread-26814) 0: Using GPU: 0
No server factory registered for the given ServerDef: cluster {
  job {
    name: "worker"
    tasks {
      key: 0
      value: "gpu05.cluster:40727"
    }
  }
}
job_name: "worker"
protocol: "grpc_rdma"

And it keeps on repeating. Does that imply that I have properly configured TensorFlowOnSpark for GPUs but it just can not find them when running the jobs, for whatever reason?

[leewyang]

Can you try removing the --rdma argument to see if it gets further? It looks like the RDMA support might not be properly compiled. If so, will need help from @junshi15 (who's out this week).