MNIST example at one out of four worker nodes only about caffeonspark

[junshi15]

It's a yarn problem. Yarn scheduled all workers on the same physical node. You need to tell yarn stop doing that.

One trick we used was executor memory. Let's say each node has 16GB memory, you can set the following in your spark-submit.

--config spark.executor.memory = 9g

Now, no two workers can be on the same node, otherwise, you need 18GB memory. Of course, there will be only 7GB left on each node (maybe less due to overhead).

[romanzac]

It worked for me. Thanks a lot.
R.