Poor efficiency while running many worker instance nodes about caffeonspark

[anfeng]

@dejunzhang What's is the network bandwidth b/w your nodes? It's very likely your network is congested. It will be interesting to get some network usage info.

May I assume that both runs use identical batch size? Batch size is for per worker instance. You should adjust batch size by dividing it by $(SPARK_WORKER_INSTANCES).

[junshi15]

Several factors could contribute to the slowness:

1. weakest node in the cluster. Training is done synchronously, the slowest node brings down the over speed.

2. network bandwidth. Two drawbacks with ethernet connections: (a) GPU memory needs to be copied to host memory before transmission. The copying can be slow. (b) Ethernet is usually slow. We use Infiniband and RDMA for fast memory and network access.

3. as @anfeng mentioned, your effective batch size doubles when you use two nodes since the batch size in the prototxt file is per device.

[dejunzhang]

@anfeng the slave nodes are connected with a 1Gb/s switch. Yes, I didn't modify the batch size just now, and they are the same as in single node. I will have a try. thanks.
@junshi15 the slave nodes have the same hardware and environment. Now we only have ethernet connections between nodes.

[mriduljain]

@dejunzhang Let us know if you see differences with above config changes etc, whenever

[dejunzhang]

@mriduljain the network usage during the trainning is about 90 ~ 300Mb/s and the time cost is slight higher than the single node but with lower loss. Thank you.