dmlc / dmlc-core Goto Github PK
View Code? Open in Web Editor NEWA common bricks library for building scalable and portable distributed machine learning.
License: Apache License 2.0
A common bricks library for building scalable and portable distributed machine learning.
License: Apache License 2.0
Guys:
I am trying to launch example traing_mnist.py on Yarn cluster which with 50 nodes, the launch script like below:
${mxnethome}/tools/launch.py -n 2 \
--launcher yarn \
python ${mxnethome}/example/image-classification/train_mnist.py --network lenet --kv-store dist_async
It keeps failed and the yarn container told me below logs:
File "./train_mnist.py", line 1, in <module>
import find_mxnet
ImportError: No module named find_mxnet
and there is no method to add -file
option to the script of tools/launch.py?
Any one run the distributed training example on Yarn successfully?
runtime error log:
src/io/input_split_base.cc:117: FILE size not calculated correctly
Yarn container error:
15/05/12 15:44:28 INFO dmlc.ApplicationMaster: [DMLC] Task 2 exited with status 250 Diagnostics:Exception from container-launch.
Container id: container_1431414776876_0010_01_000004
Exit code: 250
Stack trace: ExitCodeException exitCode=250:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Container exited with a non-zero exit code 250
dmlc will throw exception on hadoop cluster using LCE which uses nobody to run tasks:
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:290)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Shell output: main : command provided 1
main : user is nobody
main : requested yarn user is dmlc
I have tested using your package for mapping soil types (geographical data). I was first simply using 'factor' format for the train.y
, but then realized that predict will also add class '0'
(basically missing values). After I converted factors to integers (0,1,2,..,p) the maps were fine.
I have installed R package from source code. It is a pity that it is not also available via CRAN. Otherwise, thanks for such a great package!
const int nthread = omp_get_max_threads();
data->resize(nthread);
...
#pragma omp parallel num_threads(nthread_)
{
// threadid
int tid = omp_get_thread_num();
size_t nstep = (chunk.size + nthread - 1) / nthread;
...
}
Why the num_threads is nthread_
, buy use nthread
to calculate nstep?
In my machine, nthread_ = 2
and nthread = 20
, so I only can read 1/10 data, 18 data blocks are empty ...
Why not only use one of them? @tqchen
currently libsvm_parser parses each line twice, which can be improved, and handling of isnumberchars() is awkard. Fix them when you have time.
I've had instances where I would like to first generate a JSON object in one function and then embed this into a array in another JSON object.
In such cases, I do something like:
dmlc::JSONWriter writer(&os);
writer.BeginArray();
for (size_t i = 0; i < dump.size(); ++i) {
if (i != 0) writer.WriteArraySeperator();
os << dump[i]; // Dump the previously generated JSON here
}
writer.EndArray();
But this means the indenting is not right anymore. Would be nice to be able to fix the indenting with:
dmlc::JSONWriter writer(&os);
writer.BeginArray();
for (size_t i = 0; i < dump.size(); ++i) {
if (i != 0) writer.WriteArraySeperator();
os << dmlc::JSONWriter.AddIndent(dump[i], 1); // Dump the previously generated JSON here
}
writer.EndArray();
or something similar
See https://github.com/dmlc/wormhole/blob/master/learn/linear/sgd/ftrl.cc#L172 the glog was explicitly included in the bootstrap call. I think we will need a redirection function in dmlc/logging.h
So all loggings goes from dmlc/logging.h and then redirected to glog
I submit job via
tracker/dmlc-submit \
--cluster yarn \
--num-workers 1 \
--num-servers 1 \
--queue my_queue \
--worker-cores 4 \
--server-cores 4 \
--ship-libcxx /opt/gcc-4.8.2/lib64/ \
bin/linear.dmlc demo/linear/conf.linear.train
But the application is killed because of exceeding allocated virtual memory. The full log listed below.
16/09/01 14:48:03 INFO impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1472556645971_451712_01_000003
16/09/01 14:48:03 INFO impl.ContainerManagementProtocolProxy: Opening proxy : rz-data.rz.xxx.com:8043
16/09/01 14:48:03 INFO dmlc.ApplicationMaster: onContainerStarted Invoked
16/09/01 14:48:03 INFO dmlc.ApplicationMaster: onContainerStarted Invoked
16/09/01 14:48:13 INFO dmlc.ApplicationMaster: [DMLC] Task 0 killed because of exceeding allocated virtual memory
16/09/01 14:48:13 INFO impl.NMClientAsyncImpl: Processing Event EventType: STOP_CONTAINER for Container container_1472556645971_451712_01_000002
16/09/01 14:48:13 INFO impl.NMClientAsyncImpl: Processing Event EventType: STOP_CONTAINER for Container container_1472556645971_451712_01_000003
16/09/01 14:48:13 INFO impl.ContainerManagementProtocolProxy: Opening proxy : rz-data.rz.xxx.com:8043
16/09/01 14:48:13 INFO impl.ContainerManagementProtocolProxy: Opening proxy : rz-data.rz.xxx.com:8043
16/09/01 14:48:13 INFO dmlc.ApplicationMaster: onContainerStopped Invoked
16/09/01 14:48:13 INFO dmlc.ApplicationMaster: onContainerStopped Invoked
16/09/01 14:48:13 INFO dmlc.ApplicationMaster: Application completed. Stopping running containers
16/09/01 14:48:13 INFO dmlc.ApplicationMaster: Diagnostics., num_tasks2, finished=0, failed=2
[DMLC] Task 0 killed because of exceeding allocated virtual memory
16/09/01 14:48:13 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
Exception in thread "main" java.lang.Exception: Application not successful
at org.apache.hadoop.yarn.dmlc.ApplicationMaster.run(ApplicationMaster.java:290)
at org.apache.hadoop.yarn.dmlc.ApplicationMaster.main(ApplicationMaster.java:115)
End of LogType:stderr
How to solve this problem?
tracker/dmlc_tracker/mpi.py does not transfer LD_LIBRARY_PATH to slave nodes variable unlike SSH launcher. When the libraries such as CUDA, MKL are used, the other nodes may raise the error.
dmlc/dmlc-core. I'm coming
The introduce is important for the new people,say how to use it.
like the fis, fis is baidu product.
http://www.imooc.com/learn/220
@Earthson
I cloned the code today, and test the dmlc-submit, but find this error
sh run_xgboost.sh
Traceback (most recent call last):
File "/opt/meituan/xgboost/dmlc-core/tracker/dmlc-submit", line 7, in
from dmlc_tracker import submit
File "/opt/meituan/xgboost/dmlc-core/tracker/dmlc_tracker/submit.py", line 6, in
from . import local
File "/opt/meituan/xgboost/dmlc-core/tracker/dmlc_tracker/local.py", line 10, in
from . import tracker
File "/opt/meituan/xgboost/dmlc-core/tracker/dmlc_tracker/tracker.py", line 149
if e.errno in {98, 48}:
^
SyntaxError: invalid syntax
several suggestions to better support minibatch training:
and similarly RowblockContainer.GetBlock(10, 20).
but now we need to change the codes a little bit for nonzero offset[0], one example d2af94b
it's not necessary to change the implementation, namely use strtoull to parse the string even use unsigned index type.
Need CFLAGS for non-system glog.
The glog may have different versions, e.g. one from system, one in dmlc/ps-lite @mli when using dmlc/cxxnet with ps.
dmlc-core/tracker/dmlc-submit --cluster ssh --num-workers 2 xgboost demo/binary_classification/mushroom.conf data=demo/data/agaricus.txt.train eval=demo/data/agaricus.txt.test
Traceback (most recent call last):
File "dmlc-core/tracker/dmlc-submit", line 9, in <module>
submit.main()
File "/Users/nanzhu/code/xgboost/dmlc-core/tracker/dmlc_tracker/submit.py", line 50, in main
raise RuntimeError('Unknown submission cluster type %s' % args.cluster)
RuntimeError: Unknown submission cluster type ssh
It is not always easy to have all the required arguments in the file URI spec.
Hence, it would be nice to be able to pass an additional set of arguments in the Parser::Create() funciton.
Here is the suspected code snippets:
int main(int args, char *argv[]) {
if (argc < 2) {
printf("Usage: \n");
return 0;
}
dmlc::Stream *fs = dmlc::Stream::Create(argv[1], "w");
dmlc::ostream os(fs);
os << "hello-world" << 1e-10 << std::endl;
delete fs;
}
fs is a pointer. When os is destructed, it call its member variable stream_, which stream_ = fs.
So if we delete fs before os is destructed, as above code shows, it will case dangling pointer.
I suggest delete fs in dmlc::ostream's destructor.
source code version: e9b5cdb
OS: Windows 10 x64
1>------ Build started: Project: dmlc, Configuration: Release x64 ------
1> data.cc
1>C:\Program Files (x86)\Windows Kits\10\Include\10.0.10150.0\ucrt\stdio.h(1419): warning C4005: 'vsnprintf': macro redefinition
1> C:\work\dmlc-core\include\dmlc/base.h(88): note: see previous definition of 'vsnprintf'
1>C:\Program Files (x86)\Windows Kits\10\Include\10.0.10150.0\ucrt\stdio.h(1421): fatal error C1189: #error: Macro definition of vsnprintf conflicts with Standard Library function declaration
1> io.cc
1>C:\Program Files (x86)\Windows Kits\10\Include\10.0.10150.0\ucrt\stdio.h(1419): warning C4005: 'vsnprintf': macro redefinition
1> C:\work\dmlc-core\include\dmlc/base.h(88): note: see previous definition of 'vsnprintf'
1>C:\Program Files (x86)\Windows Kits\10\Include\10.0.10150.0\ucrt\stdio.h(1421): fatal error C1189: #error: Macro definition of vsnprintf conflicts with Standard Library function declaration
1> input_split_base.cc
1>C:\work\dmlc-core\include\dmlc/logging.h(203): warning C4297: 'dmlc::LogMessageFatal::~LogMessageFatal': function assumed not to throw an exception but does
1> C:\work\dmlc-core\include\dmlc/logging.h(203): note: destructor or deallocator has a (possibly implicit) non-throwing exception specification
1> line_split.cc
1>c:\work\dmlc-core\include\dmlc./logging.h(203): warning C4297: 'dmlc::LogMessageFatal::~LogMessageFatal': function assumed not to throw an exception but does
1> c:\work\dmlc-core\include\dmlc./logging.h(203): note: destructor or deallocator has a (possibly implicit) non-throwing exception specification
1> local_filesys.cc
1>C:\work\dmlc-core\include\dmlc/logging.h(203): warning C4297: 'dmlc::LogMessageFatal::~LogMessageFatal': function assumed not to throw an exception but does
1> C:\work\dmlc-core\include\dmlc/logging.h(203): note: destructor or deallocator has a (possibly implicit) non-throwing exception specification
1> recordio_split.cc
1>c:\work\dmlc-core\include\dmlc./logging.h(203): warning C4297: 'dmlc::LogMessageFatal::~LogMessageFatal': function assumed not to throw an exception but does
1> c:\work\dmlc-core\include\dmlc./logging.h(203): note: destructor or deallocator has a (possibly implicit) non-throwing exception specification
1> recordio.cc
1>C:\Program Files (x86)\Windows Kits\10\Include\10.0.10150.0\ucrt\stdio.h(1419): warning C4005: 'vsnprintf': macro redefinition
1> C:\work\dmlc-core\include\dmlc/base.h(88): note: see previous definition of 'vsnprintf'
1>C:\Program Files (x86)\Windows Kits\10\Include\10.0.10150.0\ucrt\stdio.h(1421): fatal error C1189: #error: Macro definition of vsnprintf conflicts with Standard Library function declaration
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========
Recently, my colleague change the HDFS to federation mode, then the xgboost on yarn get the following error: Diagnostics: Incomplete HDFS URI, no host: hdfs:///tmp/temp-dmlc-yarn-application_1472437483579_135758/libstdc++.so.6
.
I read the Client.java and found that in line 143 Path dst = new Path("hdfs://" + tmpPath + "/"+ path.getName());
there is no given host. So I replace this line with String fsDefaultName = conf.get(FileSystem.FS_DEFAULT_NAME_KEY); Path dst = new Path(fsDefaultName + tmpPath + "/"+ path.getName());
and run again, then I encountered java.lang.IllegalArgumentException: Wrong FS: hdfs://ss-hadoop/tmp/temp-dmlc-yarn-application_1472437483579_135926/libstdc++.so.6, expected: viewfs://ss-hadoop/
, then I read the ApplicationMaster.java and found that the FileSystem was get by the default configuration. In this case, if the value of defaultFS
is different between the Client Machine and the Container which the Application Master running on this error is inevitable. So I added 'FileSystem dfs = e.getValue().getFileSystem(conf);' before line 195 which is FileStatus status = dfs.getFileStatus(e.getValue());
. Now the distributed xgboost runs OK again.
I think there must be more elegant solution for all these problems. If you done, don't hesitate to tell me.
Curently there is no support for mesos trackers, which is another common way to submit jobs to a cluster other than YARN. It could be interesting to support job submission tracker for mesos
Hi, tianqi. I ran XGBoost on yarn. I have ran a test code:mushroom. My script's parameters is as followed:
15 $WORK_DIR/dmlc-core/tracker/dmlc-submit \
16 --cluster=yarn \
17 --num-workers=2 \
18 --num-servers=1 \
19 --worker-cores=2 \
20 --log-file="$log_file" \
21 --ship-libcxx="/data0/zeus/soft/gcc/lib64" \
22 --worker-memory=5g \
23 --queue=root.zeus.zeus \
24 --jobname=dmlc-xgboost \
25 --hdfs-tempdir=ns1/user/zeus/xgboost \
26 --log-level='DEBUG' \
27 ../xgboost \
28 $CONF \
29 data="hdfs://ns1/dw_ext/comm_rec/ctr/xgboost/test/train/agaricus.txt.train" \
30 eval[test]="hdfs://ns1/dw_ext/comm_rec/ctr/xgboost/test/test/agaricus.txt.test" \
31 model_dir="hdfs://ns1/dw_ext//comm_rec/ctr/xgboost/test/model"
When it calls launchDummyTask, it will run launcher.py without delivering any parameters. My yarn logs is as followed:
LogType: stdout
LogLength: 32
Log Contents:
Usage: launcher.py your command
For xgboost, it would be nice to be able to ignore certain columns in the config file and build a model on top of that. This would make it simple to build models with or without certain variables using a config rather than modifying the whole data file - which takes time if the data is large
This is the proposal for a unified tracker script that is used to launch all DMLC jobs. I will post all the necessary parameters here, and please respond the discussion if we want to iterate. Previously the tracker script of DMLC job submission has been separated into different python files. While this can be convenient, it also makes user hard to find a central command instruction to submit all kinds of jobs. So I am proposing a new centralized script dmlc-submit
that will submit the jobs.
This discussion thread will eventually becomes the document as well as spec for the submission script. I am hoping we can do it in one to two weeks
Let us also discuss parameters, I think we could avoid using abbreviations and enforce full names, to make submission script more clear, as in other job submission scripts like
Some of these parameters could be set from environment variables, with prefix DMLC.
--num-workers
integer, required
--worker-cores
integer, default 1 . Number of cores needed to be allocated for worker job.--server-cores
integer, default 1, Number of cores needed to be allocated for server job.--worker-memory
string, integer + [g|m], default 1g, Memory needed for worker job.--server-memory
string, integer + [g|m], default 1g, Memory needed for worker job.--mode
string, 'mpi' or 'yarn' or 'local', default to ${DMLC_SUBMIT_MODE}, submission mode.when i run command "make lint", i get following errors. Is there something i need to configure?
python scripts/lint.py dmlc "all" include src scripts
Traceback (most recent call last):
File "scripts/lint.py", line 13, in
from pylint import epylint
ImportError: No module named pylint
make: *** [lint] Error 1
A typical CSV file:
Year,Make,Model,Description,Price,Sell
1997,Ford,E350,"ac, abs, moon",3000.00,40000
We assume there are only two types of columns: real number
, such as Price, and string
, such as Model and Description. While the category column, such as Year, can be viewed either as real or as string. The parser will use one-shot encoding for string column.
A user can specify the column format:
my_file.csv&label=col[5]&str=col[0-3]&real=col[4]
where col[0-3]
presents column 0, 1, 2, and 3. All col[0,1,2,3]
, col[0,1-3]
, col[0,1-2,3]
are valid and equal.
More options:
noheader
, there is no header linesep=\t
, treat \t
as the separaterI use RedHat 6.8 for my YARN cluster nodes, and it uses Python 2.6.6. In order to get Python 2.7 in RedHat one uses Software Collections. To activate a software collection one must do scl enable python27 'python -V'
Problem is when I try to submit my Python job to YARN as follows
dmlc-submit --cluster=yarn scl enable python27 'python -V'
... it seems to eat up the quotes and produces this error (instead of the expected Python 2.7.8)
Unable to open /etc/scl/prefixes/python!
That is the same output one gets when doing following at the bash prompt on any machine
scl enable python27 python -V
I'm trying to figure out how to fool argparse into letting this through but thought I'd bring it up
In many cases, simply recompiling code against Intel compiler (ICC) and Math Kernel Library (MKL) gains 5%~15% performance improvement over GCC. So I propose to give users the choice to build dmlc-core with ICC or GCC.
Wider vectorization has been introduced in recent (?Intel only?) CPU. For example, AVX with 256-bit SIMD (for float only) since Sandy Bridge in 2012 and AVX2 with 256-bit SIMD (for both float and integer) since Haswell in 2014. Compiler options can be introduced to enable AVX or AVX2, for example USE_AVX
and USE_AVX2
.
Even the dmlc-core's building process is straight-forward enough, adding sections like "Build" and "Install" in README will pack all the necessary things together. This also makes dmlc-core more like a standalone deployable package.Cluster administrators would love to see that.
In summary,
Build
and Install
sections in README.I think I can help and run some tests on a standard cluster. But that would be after mid August.
include/dmlc/logging.h:245: [08:12:44] src/io/hdfs_filesys.cc:156: Check failed: files != NULL Error when ListDirectory /directory//train.gz terminate called after throwing an instance of 'dmlc::Error' what(): [08:12:44] src/io/hdfs_filesys.cc:156: Check failed: files != NULL Error when ListDirectory /directory//train.gz
http://doc.mapr.com/pages/viewpage.action?pageId=29658211
maybe hdfsListDirectory() does not support wildcard in directory, could we support this feature?
The current implementation will always expect files to be under an S3 folder.
Example:
s3://bucket/data.csv => Cannot read. Fails
S3://bucket/folder/data.csv => Success
Similar issue - apache/mxnet#4578
g++ -DMSHADOW_FORCE_STREAM -Wall -O3 -I/Users/yujinke/mxnet/mshadow/ -I/Users/yujinke/mxnet/dmlc-core/include -fPIC -Iinclude -msse3 -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -I/System/Library/Frameworks/Accelerate.framework/Versions/Current/Frameworks/vecLib.framework/Versions/Current/Headers/ -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSDHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 pkg-config --cflags opencv
-DMXNET_USE_NVRTC=0 -std=c++0x -o bin/im2rec tools/im2rec.cc build/src/resource.o build/src/c_api/c_api.o build/src/c_api/c_api_error.o build/src/c_api/c_predict_api.o build/src/common/mxrtc.o build/src/engine/engine.o build/src/engine/naive_engine.o build/src/engine/threaded_engine.o build/src/engine/threaded_engine_perdevice.o build/src/engine/threaded_engine_pooled.o build/src/io/io.o build/src/io/iter_csv.o build/src/io/iter_image_recordio.o build/src/io/iter_mnist.o build/src/kvstore/kvstore.o build/src/ndarray/ndarray.o build/src/ndarray/ndarray_function.o build/src/operator/activation.o build/src/operator/batch_norm.o build/src/operator/block_grad.o build/src/operator/broadcast_reduce_op.o build/src/operator/cast.o build/src/operator/concat.o build/src/operator/convolution.o build/src/operator/crop.o build/src/operator/cross_device_copy.o build/src/operator/cudnn_batch_norm.o build/src/operator/deconvolution.o build/src/operator/dropout.o build/src/operator/elementwise_binary_op.o build/src/operator/elementwise_binary_scalar_op.o build/src/operator/elementwise_sum.o build/src/operator/elementwise_unary_op.o build/src/operator/embedding.o build/src/operator/fully_connected.o build/src/operator/identity_attach_KL_sparse_reg.o build/src/operator/leaky_relu.o build/src/operator/loss_binary_op.o build/src/operator/lrn.o build/src/operator/native_op.o build/src/operator/ndarray_op.o build/src/operator/operator.o build/src/operator/operator_util.o build/src/operator/pooling.o build/src/operator/regression_output.o build/src/operator/reshape.o build/src/operator/slice_channel.o build/src/operator/softmax_activation.o build/src/operator/softmax_output.o build/src/operator/swapaxis.o build/src/operator/upsampling.o build/src/optimizer/optimizer.o build/src/optimizer/sgd.o build/src/storage/storage.o build/src/symbol/graph_executor.o build/src/symbol/graph_memory_allocator.o build/src/symbol/static_graph.o build/src/symbol/symbol.o /Users/yujinke/mxnet/dmlc-core/libdmlc.a -pthread -lm -framework Accelerate pkg-config --libs opencv
ar: no archive members specified
usage: ar -d [-TLsv] archive file ...
ar -m [-TLsv] archive file ...
ar -m [-abiTLsv] position archive file ...
ar -p [-TLsv] archive [file ...]
ar -q [-cTLsv] archive file ...
ar -r [-cuTLsv] archive file ...
ar -r [-abciuTLsv] position archive file ...
ar -t [-TLsv] archive [file ...]
ar -x [-ouTLsv] archive [file ...]
make: *** [lib/libmxnet.a] Error 1
make: *** Waiting for unfinished jobs....
clang: error: no such file or directory: '/Users/yujinke/mxnet/dmlc-core/libdmlc.a'
clang: error: no such file or directory: '/Users/yujinke/mxnet/dmlc-core/libdmlc.a'
make: *** [lib/libmxnet.so] Error 1
make: *** [bin/im2rec] Error 1
Hi, I am trying to install mxnet package for R on Ubuntu 16.04 LTS. However I am running into an issue.
These are the steps I took:
sudo apt-get install -y build-essential git libatlas-base-dev libopencv-dev git clone --recursive https://github.com/dmlc/mxnet cd mxnet; make -j$(nproc)
R CMD BUILD R-package/ R CMD INSTALL mxnet_0.7.tar.gz
But when running install I get the following error
* installing to library ‘/home/mich/R/x86_64-pc-linux-gnu-library/3.2’
* installing *source* package ‘mxnet’ ...
** libs
g++ -I/usr/share/R/include -DNDEBUG -I../inst/include -I"/home/mich/R/x86_64-pc-linux-gnu-library/3.2/Rcpp/include" -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c executor.cc -o executor.o
In file included from executor.cc:9:0:
./base.h:12:23: fatal error: dmlc/base.h: No such file or directory
compilation terminated.
/usr/lib/R/etc/Makeconf:141: recipe for target 'executor.o' failed
make: *** [executor.o] Error 1
ERROR: compilation failed for package ‘mxnet’
* removing ‘/home/mich/R/x86_64-pc-linux-gnu-library/3.2/mxnet’
Am I missing something?
Thanks
What does the following check failed mean ?
in dmlc-core/src/data/row_block.h
CHECK(offset.back() == value.size() || value.size() == 0);
And how to fix that ?
Thanks
Is there any specific reason why we didn't make it a header-only library? I think a header-only library is much easier to integrate into other projects.
see dmlc/xgboost#879 I also observe this in my experiments. This was introduced in https://github.com/dmlc/dmlc-core/blob/master/tracker/yarn/src/main/java/org/apache/hadoop/yarn/dmlc/Client.java#L273
@seusofthd Please look into it and submit a fix. I think we need to add a hook only for signal events such as ctrl +c but not normal shutdown
hi all, when I was compiling mxnet using vs2013 solution generated by CMake Gui, there were always compilation errors such as:
"mxnet\include\mxnet/ndarray.h(594): error : has_saveload is not a template"
"D:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\INCLUDE\utility(201): error : incomplete type is not allowed detected during instantiation of class "std::pair<_Ty1, _Ty2> [with _Ty1=const int, _Ty2=mxnet::NDArray]" "
"mxnet\src\optimizer\./sgd-inl.h(114): error : incomplete type is not allowed"
"mxnet\src\optimizer\./sgd-inl.h(114): error : pointer to incomplete class type is not allowed"
I used "Microsoft Visual C++ Compiler Nov 2013 CTP" files overwrite the ones exist in the vs2013 installation path. Version of the third-party libraires are Opencv3.0.0, Cuda 7.5 and cudnn is not used. Can anyone help me point out the problem? Thanks a lot!
I find this file changed 15 hours ago and this change may cause mxnet build error:
In file included from /usr/include/c++/4.8/typeindex:35:0,
from /dl/xlvector/mxnet/dmlc-core/include/dmlc/./json.h:19,
from /dl/xlvector/mxnet/dmlc-core/include/dmlc/parameter.h:22,
from include/mxnet/base.h:12,
from src/ndarray/./ndarray_function.h:11,
from src/ndarray/ndarray_function.cu:9:
/usr/include/c++/4.8/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
#error This file requires compiler and library support for the
I am trying a fresh install of cxxnet and hit the following compilation error
src/nnet/././nnet_config.h:137:29: required from here
./dmlc-core/include/dmlc/./serializer.h:61:27: error: ‘Write’ is not a member of ‘dmlc::serializer::UndefinedSerializerFor<char>’
Else::Write(strm, data);
any idea of what is going wrong ?
I was looking at lint.py and lint3.py and they're the same with very minor differences, it makes sense to reduce these to a single lint.py with if conditions for python3.
There have been efforts to do this, but it's not yet complete - f1e1033#diff-460cb56a7855c776ca8943d6f4f2c6c4
The differences are:
We really need one.
It would be nice to have an argument in the dmlc-submit script which will copy all the environment variables into the worker nodes.
This would be an easy way to do it rather than doing each env variable by hand for simple tasks.
--env-all
?
Currently supported distributed file systems include Amazon Web Services S3, Microsoft Azure, and Apache Hadoop HDFS.
Aliyun is the largest cloud computing company in China with 22.8% market share of the IaaS market in 2014 and 106% year over year growth as of June 2015.
Aliyun Object Storage Service OSS provides C SDK with easy to install dependencies and detailed step-by-step documentation.
This is a feature similar with #14.
λ EarthsonPC dmlc-core → λ git master* → python tracker/dmlc_tracker/tracker.py --num-workers 2
2016-07-17 18:03:18,522 WARNING gethostbyname(socket.getfqdn()) failed... trying on hostname()
Traceback (most recent call last):
File "tracker/dmlc_tracker/tracker.py", line 145, in __init__
sock.bind((hostIP, port))
OSError: [Errno 48] Address already in use
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "tracker/dmlc_tracker/tracker.py", line 475, in <module>
main()
File "tracker/dmlc_tracker/tracker.py", line 470, in main
start_rabit_tracker(args)
File "tracker/dmlc_tracker/tracker.py", line 432, in start_rabit_tracker
rabit = RabitTracker(hostIP=get_host_ip(args.host_ip), nslave=args.num_workers)
File "tracker/dmlc_tracker/tracker.py", line 149, in __init__
if e.errno in 98:
TypeError: argument of type 'int' is not iterable
Exception ignored in: <bound method RabitTracker.__del__ of <__main__.RabitTracker object at 0x101f9b0b8>>
Traceback (most recent call last):
File "tracker/dmlc_tracker/tracker.py", line 163, in __del__
self.sock.close()
AttributeError: 'RabitTracker' object has no attribute 'sock'
tracker.py only check error code 98 to reselect a new port, I think code 98 should also be considered:)
I had make the changes according to #10 , but it still come out with the error as following:
/usr/bin/ld: /usr/lib/hadoop/lib/native/libhdfs.a(hdfs.c.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/lib/hadoop/lib/native/libhdfs.a: could not read symbols: Bad value
//
instead of /
Submitting using the dmlc-submit script is resulting in the successful spawning and completion of the job on yarn, but the dmlc-submit.py script is waiting indefinitely. The stack trace generated by killing at that point is resulting in the following stack trace :
File "./tracker/dmlc-submit", line 9, in
submit.main()
File "/idn/home/vdevabat/workingDir/tracker/dmlc_tracker/submit.py", line 46, in main
yarn.submit(args)
File "/idn/home/vdevabat/workingDir/tracker/dmlc_tracker/yarn.py", line 123, in submit
pscmd=(' '.join([YARN_BOOT_PY] + args.command)))
File "/idn/home/vdevabat/workingDir/tracker/dmlc_tracker/tracker.py", line 413, in submit
rabit.join()
File "/idn/home/vdevabat/workingDir/tracker/dmlc_tracker/tracker.py", line 329, in join
self.thread.join(100)
File "/usr/lib64/python2.6/threading.py", line 655, in join
self.__block.wait(delay)
File "/usr/lib64/python2.6/threading.py", line 258, in wait
_sleep(delay)
Env : Mapr version of hadoop 2.5.1 is being used. The error is reproducible on the basic.cc file provided as a part of documentation. Any help appreciated.
I use ImageRecordIter to access the val.rec
in hdfs, like val =mx.io.ImageRecordIter(path_imgrec = "hdfs://address:54310/data_dir/val.rec",
however the error log is :Exception in thread "main" java.io.IOException: Incomplete HDFS URI, no host: hdfs://address:54310:0
I track the code and it seems whether the address has a port is ignored, and a port 0 is added. Is this a bug?
this is the related issue i found:
#81
CDH is slightly difference to the native hadoop. We can support it or write a document to let users change the things accordingly.
hdfs.h
hdfs.a
check the version, on my case:
~/hadoop version
Hadoop 2.3.0-cdh5.1.0
the solution:
according to my version, i download
http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.3.0-cdh5.1.0-src.tar.gz
extract, and copy
hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
to the fold include/
.
make/dmlc.mk
to use the .a versionDMLC_LDFLAGS+= $(HADOOP_HDFS_HOME)/lib/native/libhdfs.a -L$(LIBJVM) -ljvm -Wl,-rpath=$(LIBJVM)
It should compile now. Then we need to set the environment properly (I guess it is also necessary for the the native hadoop). In my case, I set .bashrc
export HADOOP_HOME=/usr/lib/hadoop
export HADOOP_CONF=/etc/hadoop/conf
export CLASSPATH=${HADOOP_CONF}:$(find ${HADOOP_HOME} -name *.jar | tr '\n' ':')
And there is warning in 64bit centos
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
but it seems OK to ignore it according to this
Support for google cloud storage can be done similarly as https://github.com/dmlc/dmlc-core/blob/master/src/io/s3_filesys.cc via rest API
It could involve a refactor of s3 filesys to be a general base, which take s3 and gcs as child class. Guess similar thing could also be done to support Azure's cloud.
Anyone who is interested in claim this is welcomed to reply on this thread
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.