yarn-examples-simple-command failed on cdh5@centos about spring-hadoop-samples

[jvalkeal]

Might be something to do with build because samples are still on vanilla hadoop 2.2.0 and cdh5 might be newer. Did you found container logs(stdout and stderr logs for application master) which might tell more. You could also try our IO guides https://spring.io/guides?filter=yarn. Building Spring YARN Projects with Gradle/Maven guides have instructions howto build against different distros and cdh5 is one supported.

[chang-chao]

Finally,I made it work.(Though still there is something that I cannot understand)

There was ClassNotFound(log4j,org.apache.hadoop.conf.Configuration) exception in stderr(Appmaster.stderr) for application master log,So I commented out the "hadoopruntime.exclude" declaration in build.gradle,after that the app finished and succeeded.

By the way,the following should be noticed.
1.the following env viariables ,which seems to be used when master is started,are not set automatically when cdh is installed,I had to set them manually.

• HADOOP_CONF_DIR
• HADOOP_COMMON_HOME
• HADOOP_HDFS_HOME
• HADOOP_MAPRED_HOME
• HADOOP_YARN_HOME

2.As with this cloudera issue:Container erases temporary file and shell script immediately after execution,you have to monitor file to get the log file.

[chang-chao]

After digging nearly two days,I think I found the reason why ClassNotFound(class in commons-logging lib) exception occured in the simple yarn app.

Firstly,we should definitely set the yarn.application.classpath value (through adding yarn-site.xml to the classpath of the client) according to cluster env in the app client .If not so,the default value(eg.$HADOOP_COMMON_HOME/share/hadoop/common/*,...) will be used,which is not the right path in many distros(at least CDH5)

Secondly,Don't use multiple-line values for "yarn.application.classpath" ,in yarn-site.xml,in my cluster (cdh5,java7,centos),classpath could not be fully and correctly parsed if we did so.
unfortunately the yarn-site.xml in cdh5 cluster is containing the multi-line value,which I think is a bug.