Comments (8)
We will release some numbers.
from byteps.
@MarvinLong
I've test bert in byteps. The result show byteps can actually improve the training speed.
I test it in a single machine with 4 V100 gpus.
horovod-bert: global_step/sec: 1.03465 examples/sec: 33.1087
byteps: global_step/sec: 1.56383 examples/sec: 50.0426 (Average of these data)
from byteps.
@MarvinLong Thank you for sharing this!
Would you mind sharing more information? I have a couple of questions:
- What is the training speed on single V100 GPU (no communication)?
- Is your scripts (horovod & byteps) for bert training open-sourced? If yes, could you please provide the link so that we can also try to reproduce?
- What is the batch size that you use per GPU?
from byteps.
@ymjiang
1. horovod: global_step/sec: 1.68363 examples/sec: 53.8761
byteps: I'm not sure if my code problem, it has bug on single v100. If I use more than one v100,
there will be no problem.
2. Sorry, I can't upload files at company
3. text_length=256, batch_size=32
from byteps.
@NlpRookie Thank you very much for sharing. We will also keep up reproducing these results.
from byteps.
@NlpRookie
Hi, thank you for sharing that. Your result is faster than mine, when I trained a bert(vocab size=100k) with 4 v100(16G) gpus, I got the speed (31.3 samples/s).
But what I actually care about is the preformance of distributed training. Now I have 4 machines with IB connected, and no other cpu machines. So I can only test the speed with 2 machines (16 gpus), the other 2 must be the server.
Will you test this with distributed training?
from byteps.
@NlpRookie
But https://github.com/bytedance/byteps/blob/master/docs/best-practice.md said
In non-distributed mode, BytePS is basically doing NCCL allreduce, so it will not outperform Horovod/NCCL much. BytePS implemented priority-based scheduling, which may improve the training speed by 0%~15%, depending on your training task.
Your result shows you got 1.5 speed up ratio.
from byteps.
@MarvinLong
I got the result with testing this model https://github.com/google-research/bert
- BERT-Base, Chinese: Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M parameters
And i test it with distributed training , got the speed (47.251 samples/s) for 1 worker.(I have no more machines)
from byteps.
Related Issues (20)
- Stuck in the bps.init(). HOT 7
- Is it right to do allreduce immediately for non-zero ranks in bytescheduler? HOT 2
- 啥时候支持sparse模型?
- 有计划支持纯cpu吗?我们worker也用cpu机器的 HOT 2
- benchmark with cross barrier error
- Successfully installed BytePS but cannot import byteps.torch or byteps.tensorflow HOT 2
- Running multiple workers on a single GPU machine
- Release BytePS docker image support for TF2
- 安装报错 HOT 1
- Communication failure in MXNet with BytePS HOT 3
- support for fault tolerance and straggler mitigation
- broadcast and is_initialized api are not supported with pytorch.
- Supported environment
- 安装问题
- Mistakes of Workload calculation HOT 5
- How does the tensorflow scheduler plugin used in the tf_benchmark_cnn.py HOT 1
- segmentation fault while launching the worker HOT 1
- Is there any benchmark comparison with Megatron-LM ?
- 支持的cuda和pytorch版本
- install failed
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from byteps.