Comments (11)
TODO:
- Get to the bottom of mysterious netty NoClassDefFound exceptions in circle builds. All of the tests that fail with this exception pass locally, which is quite confusing. Error seems to always come from initializing
MiniDFSCluster
For example:
java.lang.NoClassDefFoundError: io/netty/channel/ChannelFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:905)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1295)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:481)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2601)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2489)
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1497)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:847)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:483)
-
Make sure that new dist layout is valid.
-
Run hadoop-aws tests and make sure they all pass
from hadoop.
Can see it downloading netty-all in the circle container logs:
Downloading: http://dl.bintray.com/palantir/releases/io/netty/netty-all/4.0.23.Final/netty-all-4.0.23.Final.pom
A look at that JAR suggests that it should be looking for io/netty/bootstrap/ChannelFactory
instead:
srand-mbp:4.0.23.Final srand$ jar tf netty-all-4.0.23.Final.jar | grep ChannelFactory
io/netty/bootstrap/ChannelFactory.class
io/netty/bootstrap/AbstractBootstrap$BootstrapChannelFactory.class
from hadoop.
More info: IntelliJ says that the ServerSocketChannel
referenced in line 905 of DataNode
(mentioned in the stack trace) is a java.nio.channels.ServerSocketChannel
, not anything from netty.
from hadoop.
Suspecting 0bba8d7 might be related.
from hadoop.
Additionally hadoop-mapreduce doesn't even compile on circle, but it does compile locally. Something is seriously askew with the way that circle is running the tests.
from hadoop.
The latter issue likely has to do with e51312e. Presumably I botched the merge somehow, though nothing about the merge commit looks off to me.
from hadoop.
Would appear to have been a caching issue with circle based on how https://circleci.com/gh/palantir/hadoop/176 was doing before I cancelled it. Workaround for now is to just mvn clean install
in each container before running any tests.
from hadoop.
New dist layout looks good, and the mystery of the missing netty class has been solved. Just need to:
- Run hadoop-aws tests and make sure those check out.
- Check test results for rc5 (currently building) and make sure there are no surprises.
Then can release.
from hadoop.
Test results for rc5 are kind of scary, but no more scary than other test runs. The tests just seem to be a mess general -- would have to go on an excluding rampage to get a truly green run.
A bit more worrisome is that there's one test failure in the s3a integration test suite -- ITestS3AContractRootDir.testRecursiveRootListing
is failing with a timeout when trying to call AmazonS3Client#listObjects
. Trying to determine whether this is an actual problem or not.
from hadoop.
Test is also failing in the same way on Apache branch-2.8.0 -- calling it good as far as this release is concerned.
from hadoop.
Kicked off the build; closing.
from hadoop.
Related Issues (20)
- tell maven to be quiet during mvn install
- eventually stop ignoring these tests HOT 1
- consider backporting HADOOP-14204 HOT 2
- Publish source jars HOT 1
- watch for 2.8.1 release HOT 3
- backport HDFS-11538 HOT 1
- rebase off of 2.8.1 HOT 2
- track new s3a stuff (other than s3guard) HOT 3
- use semver HOT 1
- stop ignoring test failures during CI HOT 1
- Tracking test failures on branch-2.8.2 HOT 1
- Can we publish sources? HOT 3
- backport adl fixes
- Revert our backport of HADOOP 14475 HOT 4
- probably backport HDFS-13702 HOT 1
- fix or ignore failing tests HOT 1
- Link tickets in Palantir Changelog HOT 1
- build dist with native libs
- publish layout is broken
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hadoop.