Comments (5)
So, after some digging we found out the reason it was slow.
Distributed XGBoost with MPI is copying the data back and forth across the two machines and that is making the whole computation slow-down.
Anybody has any ideas on how to fix this data copying issue?
Thanks,
Ankur
from wormhole.
The data is indeed loaded from distributed data store, but only at startup time. So you can tell the difference from longer number of rounds.
The major goal of distributed xgboost is to scale up to the scale that could not be handled by single machine version. So it is totally possible that distributed version running slower than single node version, if the data fits into single node.
from wormhole.
Hi Tianqi,
Thank you for your response!
So, if I understand you correctly, speed would be of secondary concern as long as distributed xgboost can scale up across machines. It is good to understand the design goal, since that makes clear the trade-offs that have been made in the development.
Having said that, do you have any ideas on how it might be possible to speed up the distributed implementation of xgboost? In your opinion, would moving to Hadoop framework be beneficial here for speedup as compared to the MPI framework, in other words, does the xgboost implementation on top of Hadoop also loads data from a distributed data store over the network?
Thanks,
Ankur
from wormhole.
Hi @ankurd28 Speed is definitely important for us.
As the data scales up, the data loading cost over network is minor compared to the running cost of training in our experience (This is different from data processing problems like mapreduce, where little computation is done on each examples, and data locality is crucial).
Because more computation hits in as we get more data. It is likely not a problem for larger dataset. For small dataset, however, as the running cost already was low, and the data loading bottleneck surface up.
from wormhole.
Hi Tianqi,
Thanks a lot for your response!
I completely understand your point!
Best,
Ankur
from wormhole.
Related Issues (20)
- xgboost on yarn seems getting stuck HOT 8
- document xgboost YARN build parameters HOT 4
- ./build.sh
- difacto.dmlc hanged in nanosleep() HOT 2
- RAM not scaling HOT 12
- wormhole no longer makes HOT 5
- Conflict between dynamic and static glog linkages in OS-X HOT 4
- linear-dmlc: segmentation fault for large train file
- Python/R Interface to wormhole
- get errors running on local machine HOT 1
- Check failed at the end of linear trainer in YARN mode
- issue for the run_yarn.sh HOT 1
- Error in building wormhole HOT 1
- [question] Asynchronous SGD?
- what is relationship between the kv_app(KVWorker, KVServer) and Apps(BCD, LBFGS, SGD)
- Train on apache hadoop yarn takes more time as the worker >=2 in configuration
- [DMLC] Task 0 killed because of exceeding allocated virtual memory
- Can we change the thread number while running the application?
- linear local mode error:JUST_A_UNKNOWN_NODE is disconnected
- linear/difacto coredump
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wormhole.