release 2.8.0-palantir3 about hadoop

[sjrand]

TODO:

• Get to the bottom of mysterious netty NoClassDefFound exceptions in circle builds. All of the tests that fail with this exception pass locally, which is quite confusing. Error seems to always come from initializing MiniDFSCluster

For example:

java.lang.NoClassDefFoundError: io/netty/channel/ChannelFactory
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:905)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1295)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:481)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2601)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2489)
	at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1497)
	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:847)
	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:483)

• Make sure that new dist layout is valid.

• Run hadoop-aws tests and make sure they all pass

[sjrand]

Can see it downloading netty-all in the circle container logs:

Downloading: http://dl.bintray.com/palantir/releases/io/netty/netty-all/4.0.23.Final/netty-all-4.0.23.Final.pom

A look at that JAR suggests that it should be looking for io/netty/bootstrap/ChannelFactory instead:

srand-mbp:4.0.23.Final srand$ jar tf netty-all-4.0.23.Final.jar | grep ChannelFactory
io/netty/bootstrap/ChannelFactory.class
io/netty/bootstrap/AbstractBootstrap$BootstrapChannelFactory.class

[sjrand]

More info: IntelliJ says that the ServerSocketChannel referenced in line 905 of DataNode (mentioned in the stack trace) is a java.nio.channels.ServerSocketChannel, not anything from netty.

[sjrand]

Suspecting 0bba8d7 might be related.

[sjrand]

Additionally hadoop-mapreduce doesn't even compile on circle, but it does compile locally. Something is seriously askew with the way that circle is running the tests.

[sjrand]

The latter issue likely has to do with e51312e. Presumably I botched the merge somehow, though nothing about the merge commit looks off to me.

[sjrand]

Would appear to have been a caching issue with circle based on how https://circleci.com/gh/palantir/hadoop/176 was doing before I cancelled it. Workaround for now is to just mvn clean install in each container before running any tests.

[sjrand]

New dist layout looks good, and the mystery of the missing netty class has been solved. Just need to:

• Run hadoop-aws tests and make sure those check out.
• Check test results for rc5 (currently building) and make sure there are no surprises.

Then can release.

[sjrand]

Test results for rc5 are kind of scary, but no more scary than other test runs. The tests just seem to be a mess general -- would have to go on an excluding rampage to get a truly green run.

A bit more worrisome is that there's one test failure in the s3a integration test suite -- ITestS3AContractRootDir.testRecursiveRootListing is failing with a timeout when trying to call AmazonS3Client#listObjects. Trying to determine whether this is an actual problem or not.

[sjrand]

Test is also failing in the same way on Apache branch-2.8.0 -- calling it good as far as this release is concerned.

[sjrand]

Kicked off the build; closing.