Comments (5)
Here are some performance tuning tips: #68 (comment).
For your case, you need to use more servers. For example, you can use 2 physical machines and put 4 server instances on each physical machine (8 server instances in total). Since you have many identical nodes, I think the resources are quite sufficient for you. Note that the scheduler does not affect the training performance, so you can put it anywhere you like (e.g., colocate it with servers).
from byteps.
No matter how I change the properties of each node, the computational efficiency has not improved. I checked that the communication only supports 1GBytes. Is the bottleneck of communication affecting efficiency? Thank you
from byteps.
It seems that your network bandwidth is too low. In our tests, the bandwidth is about 20Gbps.
Is it possible for you to try larger bandwidth?
from byteps.
@CIDWLY It seems that you only have 1Gbps network interface card hardware. In that case, BytePS obviously can't help you get >1Gbps network bandwidth.
However, BytePS can help you better utilize the bandwidth you have than other distributed training implementation, like MPI. If you compare BytePS with, e.g., Horovod, you should see the advantage.
from byteps.
There are some difficulties in upgrading our bandwidth. As far as you know, what models with small enough parameters can make full use of GPU in our current experimental environment? Or is there a parameter compression method applicable in BytePs?
Thank you
from byteps.
Related Issues (20)
- Stuck in the bps.init(). HOT 7
- Is it right to do allreduce immediately for non-zero ranks in bytescheduler? HOT 2
- 啥时候支持sparse模型?
- 有计划支持纯cpu吗?我们worker也用cpu机器的 HOT 2
- benchmark with cross barrier error
- Successfully installed BytePS but cannot import byteps.torch or byteps.tensorflow HOT 2
- Running multiple workers on a single GPU machine
- Release BytePS docker image support for TF2
- 安装报错 HOT 1
- Communication failure in MXNet with BytePS HOT 3
- support for fault tolerance and straggler mitigation
- broadcast and is_initialized api are not supported with pytorch.
- Supported environment
- 安装问题
- Mistakes of Workload calculation HOT 5
- How does the tensorflow scheduler plugin used in the tf_benchmark_cnn.py HOT 1
- segmentation fault while launching the worker HOT 1
- Is there any benchmark comparison with Megatron-LM ?
- 支持的cuda和pytorch版本
- install failed
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from byteps.