node.js mapreduce library that has concept of "once write run anywhere" for hadoop framework.
##Motivation Scalable computing is more and more important for analyse and aggregate BigData, and Map-Reduce is scalable computing algorithm for distributed platform used in Hadoop or more another service.
It is important for think about scalable programing to stable running for increasing data day by day. But case of "Map-Reduce is exaggerated for current data size (MB~GB order), but ensure scalability." are very often request.
I solve these problems to run local Map-Reduce aggregation that have portablility to Hadoop platforms.
##Install
npm i glue-mapreduce
TODO: publish
##How to use ###Script Important: This script runs on local or "Hadoop", so you should write the core of your Map-Reduce algorithm. Do not write any other algorithms.
var mr = new (require('glue-mapreduce'))();
// regist "local" input data
mr.input = function (callback) {
var error = null;
// data must be iterable
var data = fs.readFileSync('somefile.txt').toString().split('\n');
return callback(error, data);
};
// regist mapper
mr.mapper = function (mapLine, callback) {
// mapper called per iteration of input data
var error = null;
var split = mapLine.split(' ');
var key = split[0],
val = split[1];
return callback(error, [{k: key, v: val}]};
// callback can be return multi key-value pairs
};
// regist reducer
mr.reducer = function (key, values, callback) {
// reducer called per iteration of keys
return callback(error, [{k: key, v: values.length}]);
// callback can be return multi key-value pairs
};
// run Map-Reduce job
mr.run(data, function (results) {
// this callback do not called on Hadoop
/*
results is array of key value pair
[{
k: 'key',
v: 100 (reduced values)
}, ...]
*/
});
###run options glue-mapreduce make a decision about whether to run with local Map-Reduce or Hadoop streaming mapper or reducer by command-line argument.
To run local Map-Reduce, node somemapreduce.js local
or no arguments.
To run Hadoop Streaming Mapper, hadop jar hadoop-streaming.jar -mapper 'somemapreduce.js mapper' ...
To run Hadoop Streaming Reducer, hadoop jar hadoop-streaming.jar -reducer 'somemapreduce.js reducer' ...
Important: To quote command need to assign argument.
These behavior also can control by mr.mode
variable. This valiable can be taken 'local'
, 'mapper'
or 'reducer'
. e.g. mr.mode = 'local'
runs local Map-Reduce aggregation.
###Testing If you want to test your script, following command can be test hadoop mode.
node myscript.js map < myinput.txt | sort | node myscript.js red
###Contribute
####Testing
npm test
####Coverage
npm run coverage