ryancole / node-webhdfs

A WebHDFS module for Node.js.

A WebHDFS module for Node.js

Notice

I no longer use HDFS and have no plans to continue development on this module. With that said, many of the more recent patches to this module have not been from myself, and instead have been through open-source contributions. If you would like me to add you as a maintainer of this repository then just open a pull request or somehow message me and I will add you.

WebHDFS

I am currently following and testing against the WebHDFS REST API documentation for the 1.2.1 release, by Apache. Make sure you enable WebHDFS in the hdfs site configuration file.

Tests

I use Mocha and should.js for unit testing. They will be required if you want to run the unit tests. To execute the tests, simply npm test, but install the requirements first.

The following environment variables are used to configure the tests:

• HDFS_USERNAME -- your username on the HDFS cluster (default: ryan)
• HDFS_NAMENODE_1 -- hostname of your primary namenode (default: localhost)
• HDFS_NAMENODE_2 -- hostname of your secondary namenode (default: localhost)
• HDFS_BASE_PATH -- directory in which to conduct tests (default: /user/$HDFS_USERNAME)

node-webhdfs's Issues

hdfs HA does not work when an active namenode completely dies

I discovered that when the active namenode process dies or the server the active namenode is living on dies completely, the HA functionality in this module will not be able to transition over to the standby namenode after HDFS is able to handle the loss and cleanly failover to the standby namenode.

The check that triggers the namenode switch in the WebHDFSClient will only ever be run if the namenode being queried returns a webhdfs response. In the scenarios of complete process or server failure, the result is a request error which will just throw an error in the program using the client.

I have a fix and should have a PR sometime soon. There were two small, but notable changes needed to handle those failover scenarios:

1. WebHDFSClient should switch namenodes in the event of both ECONNREFUSED and
   ENOTFOUND request errors in addition to the webhdfs StandbyException.
2. Since there is a very brief period between an active namenode crash and the standby namenode failover, an exponential backoff is needed to prevent WebHDFSClient from rapidly bouncing back and forth between the standby namenode and the dead active namenode and complicating the failover process.

Kerberos support for authentication

Thanks for the contribution, I am curious if there is any plan to integrate authentication support against the WebHDFS API using Kerberos?