About GPU utilization and speed about byteps

[ymjiang]

Here are some performance tuning tips: #68 (comment).

For your case, you need to use more servers. For example, you can use 2 physical machines and put 4 server instances on each physical machine (8 server instances in total). Since you have many identical nodes, I think the resources are quite sufficient for you. Note that the scheduler does not affect the training performance, so you can put it anywhere you like (e.g., colocate it with servers).

[CIDWLY]

No matter how I change the properties of each node, the computational efficiency has not improved. I checked that the communication only supports 1GBytes. Is the bottleneck of communication affecting efficiency? Thank you

[ymjiang]

It seems that your network bandwidth is too low. In our tests, the bandwidth is about 20Gbps.

Is it possible for you to try larger bandwidth?

[bobzhuyb]

@CIDWLY It seems that you only have 1Gbps network interface card hardware. In that case, BytePS obviously can't help you get >1Gbps network bandwidth.

However, BytePS can help you better utilize the bandwidth you have than other distributed training implementation, like MPI. If you compare BytePS with, e.g., Horovod, you should see the advantage.

[CIDWLY]

There are some difficulties in upgrading our bandwidth. As far as you know, what models with small enough parameters can make full use of GPU in our current experimental environment? Or is there a parameter compression method applicable in BytePs?

Thank you