start by run_yarn.sh, following error: about dmlc-core

[panhongan]

my work directory tree:

wormhole/
wormhole/dmlc-core
wormhole/repo
wormhole/repo/xgboost
wormhole/repo/rabit
wormhole/learn/xgboost/run_yar.sh (here, I start it)

[tqchen]

I have pushed some fix to dmlc-core, can you pull dmlc-core and try to make clean and make again?

Thanks

[panhongan]

my gcc version: gcc version 4.4.7

"dmlc-core" , make failed, following errors:

In file included from src/io/threaded_input_split.h:12,
from src/io.cc:13:
include/dmlc/threadediter.h: In member function ‘virtual void dmlc::ThreadedIter::BeforeFirst()’:
include/dmlc/threadediter.h:179: error: expected primary-expression before ‘[’ token
include/dmlc/threadediter.h: In member function ‘void dmlc::ThreadedIter::Init(dmlc::ThreadedIter::Producer_, bool)’:
include/dmlc/threadediter.h:273: error: expected primary-expression before ‘[’ token
include/dmlc/threadediter.h:273: error: expected primary-expression before ‘_’ token
include/dmlc/threadediter.h:273: error: ‘dptr’ was not declared in this scope
include/dmlc/threadediter.h:273: error: unable to deduce ‘auto’ from ‘’
include/dmlc/threadediter.h:273: error: expected ‘,’ or ‘;’ before ‘{’ token
include/dmlc/threadediter.h:276: error: expected primary-expression before ‘[’ token
include/dmlc/threadediter.h:276: error: unable to deduce ‘auto’ from ‘’
include/dmlc/threadediter.h:276: error: expected ‘,’ or ‘;’ before ‘{’ token
include/dmlc/threadediter.h: In member function ‘void dmlc::ThreadedIter::Init(std::function<bool(DType**)>, std::function<void()>)’:
include/dmlc/threadediter.h:290: error: expected primary-expression before ‘[’ token
include/dmlc/threadediter.h:290: error: unable to deduce ‘auto’ from ‘’
include/dmlc/threadediter.h:290: error: expected ‘,’ or ‘;’ before ‘{’ token
src/io.cc:98: error: expected ‘}’ at end of input
src/io.cc: At global scope:
src/io.cc:98: error: expected ‘}’ at end of input
make: *** [io.o] Error 1

[tqchen]

You will need to have gcc 4.6 or higher. It is possible to build with gcc 4.4 by removing -std=c++0x

Though some of the speedup threading features will not be available

[panhongan]

‘ParseState’ is not a class or namespace

can't find this class

[panhongan]

I find some issues:

1. src/config.cc
   in class TokenizeError, need to add:
   ~TokenizeError() throw() {
   }
2. in include/dmlc/config.h

typedef std::map<std::string, std::vector> InternalMap; ==》
typedef std::map<std::string, std::vector(here is a blank space) > InternalMap;

typedef typename InternalMap::const_iterator InternalMapIterator; ==>
typedef InternalMap::const_iterator InternalMapIterator;

[panhongan]

make succeed。 thanks for your help.

I found that some high-level c++ features used in your project,
it's not available by low-level gcc. so, your project may follow the mainstream gcc.

thx

[tqchen]

To enable all the features in dmlc-core, you will need to have g++4.6(preferrably g++4.8 for c++11 features). I am aware of the backward compatibility issue with earlier version of g++, we will try to support it, but with missing support for some features.

I would encourage you to upgrade your compiler(normally you only need to do it in your user space and ship the library, we will have more details latter) to take benefit of new language features that are actually not very new at all(c++11 already exists for a long time).

Tianqi

[panhongan]

error occurs:
src/io/input_split_base.cc:118: curr=61812865595,fileptr=0,fileoffset=74175438713
src/io/input_split_base.cc:124: file offset not calculated correctly

but this error doesn't occur everytime.
so can you help to fix it.

[tqchen]

interesting.. Originally, the cause of the problem was due to the fact that hdfsRead returns incomplete. In theory the problem was fixed in the recent commit.

Such problem can occur when hdfsRead returns 0 while end-of-file was not yet reached. Which seems do not meet the spec of hdfsRead in libhdfs.

One side note: I have noticed that you used quite a large file. All the dmlc input module can take a path as input argument, in which case all the files in the path will be used as input data.

So you can use many smaller files instead of one large file(which is more natural outcome of mapreduce or other distributed programs), which might help reduce the chance of such problem if this was really caused by libhdfs

[panhongan]

cool. you're right.
here, I found a bug : if worker task failed by abortJob(), the ApplicationMaster process will not be stopped. And I fix the bug. How can I contribute the code?

[hjk41]

could you create a pull request?

On Thu, May 21, 2015 at 6:06 PM, panhongan [email protected] wrote:

cool. you're right.
here, I found a bug : if worker task failed by abortJob(), the
ApplicationMaster process will not be stopped. And I fix the bug. How can I
contribute the code?

—
Reply to this email directly or view it on GitHub
#9 (comment).

HONG Chuntao
System Research Group
Microsoft Research Asia

[hjk41]

Of course you can also send us the patch if it is small. But in that case
we won't be able to see you name in the committer list, which is bad.

On Thu, May 21, 2015 at 6:14 PM, Chuntao HONG [email protected]
wrote:

could you create a pull request?

On Thu, May 21, 2015 at 6:06 PM, panhongan [email protected]
wrote:

cool. you're right.
here, I found a bug : if worker task failed by abortJob(), the
ApplicationMaster process will not be stopped. And I fix the bug. How can I
contribute the code?

—
Reply to this email directly or view it on GitHub
#9 (comment).

HONG Chuntao
System Research Group
Microsoft Research Asia

HONG Chuntao
System Research Group
Microsoft Research Asia

[panhongan]

I have pulled request. pls check it. thx.

[tqchen]

thanks