Comments (1)
edi-bice
commented on August 29, 2024
Looked into argparser (which dmlc-core and xgboost use) source code trying to figure out any possible escaping mechanism. Couldn't find any but did find out it picks up job submission node environment and recreates on each of the execution nodes. Looked at the 'scl enable' and /opt/rh/python27/enable script and realized LD_LIBRARY_PATH is the important one. PATH is also but for some reason it doesn't carry over so I now specify full path to my SCL provided python27
Following now happily runs on my YARN 2.7.1 cluster
cd xgboost
dmlc-core/tracker/dmlc-submit --cluster=yarn --num-workers=9 --worker-cores=4 /opt/rh/python27/root/usr/bin/python tests/distributed/test_basic.py
from dmlc-core.
Related Issues (20)
- WARN_NO_PARAMDOC in Doxygen is disabled HOT 1
- Building with VS2019 HOT 2
- missing parentheses in macro HOT 4
- [Feature] Logging level support HOT 4
- Memory leak in logging. HOT 4
- SSE Check Failing on android/aarch64 HOT 1
- [Feature] stack trace printing in signal handlers
- cmake option shows errror to build the app HOT 2
- Why the size of model vary with different number of workers?
- Why ref_counter of HdfsFileSystem isn't designed with atomic_int_type ？
- How Can I run example/parameter.cc?
- Crosscompilation fails due to incorrect case HOT 1
- Bug in InputSplitBase::Chunk::Append() ?
- Build standalone test executables with CMake HOT 2
- QNX: unable to determine endianness of your machine HOT 2
- can you update concurrentqueue.h according original cameron314/concurrentqueue
- Error: dmlc/json.h:695: unknown string escape \u
- Releases for version 0.4 and 0.5 are missing
- Is data load at once?memory usage is linear to pass number/
- Deprecated CMake function `test_big_endian` used
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dmlc-core.