Has anyone test this with bert training？ about byteps

[bobzhuyb]

We will release some numbers.

[NlpRookie]

@MarvinLong
I've test bert in byteps. The result show byteps can actually improve the training speed.
I test it in a single machine with 4 V100 gpus.
horovod-bert: global_step/sec: 1.03465 examples/sec: 33.1087
byteps: global_step/sec: 1.56383 examples/sec: 50.0426 (Average of these data)

[ymjiang]

@MarvinLong Thank you for sharing this!

Would you mind sharing more information? I have a couple of questions:

1. What is the training speed on single V100 GPU (no communication)?
2. Is your scripts (horovod & byteps) for bert training open-sourced? If yes, could you please provide the link so that we can also try to reproduce?
3. What is the batch size that you use per GPU?

[NlpRookie]

@ymjiang
1. horovod: global_step/sec: 1.68363 examples/sec: 53.8761
byteps: I'm not sure if my code problem, it has bug on single v100. If I use more than one v100,
there will be no problem.
2. Sorry, I can't upload files at company
3. text_length=256, batch_size=32

[ymjiang]

@NlpRookie Thank you very much for sharing. We will also keep up reproducing these results.

[MarvinLong]

@NlpRookie
Hi, thank you for sharing that. Your result is faster than mine, when I trained a bert(vocab size=100k) with 4 v100(16G) gpus, I got the speed (31.3 samples/s).
But what I actually care about is the preformance of distributed training. Now I have 4 machines with IB connected, and no other cpu machines. So I can only test the speed with 2 machines (16 gpus), the other 2 must be the server.
Will you test this with distributed training?

[MarvinLong]

@NlpRookie
But https://github.com/bytedance/byteps/blob/master/docs/best-practice.md said

In non-distributed mode, BytePS is basically doing NCCL allreduce, so it will not outperform Horovod/NCCL much. BytePS implemented priority-based scheduling, which may improve the training speed by 0%~15%, depending on your training task.

Your result shows you got 1.5 speed up ratio.

[NlpRookie]

@MarvinLong
I got the result with testing this model https://github.com/google-research/bert

• BERT-Base, Chinese: Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M parameters

And i test it with distributed training , got the speed (47.251 samples/s) for 1 worker.(I have no more machines)