node-squad's Introduction

node-squad

Run N functions in parallel. When all the functions are finished, N new functions are triggered, until the input data set is drained.

This package is based on the excellent https://github.com/kriskowal/q
It can be used for different purpose (API throttling, rate limiting, etc.).

Getting Started

Install the module with: npm install node-squad

Usage

node-squad accepts 3 input parameters :

a data set (Array)
a function taking one item of the dataset as a parameter (the function must return a result or a promise)
the configuration hash (optional)

Configuration options:

config.squadSize : the number of function to run at the same time (10 by default).
config.disableLogs : yes, you already know what it does (false by default).
config.noCopy : by default, node-squad uses a copy of the array passed as a parameter. Set this to true to use the original array instead of a copy.
config.stopOnReject : stop the squad or not when a promise is being rejected by a job(true by default).

Example :

    var Q = require('q'),
    Squad = require('node-squad');
    
    //copyright Montell Jordan
    var dataSet = ['This','is','how','we','do','it'];
    var config = {
        squadSize:3,
        disableLogs: false
    };
    
    Squad.run(dataSet,function(item){
        var jobDeferred = Q.defer();
    
        //waiting randomly
        var randomWait = Math.floor(Math.random() * 1000) + 1;
        setTimeout(function(){
            jobDeferred.resolve(item+'♪');
        },
        randomWait);
    
        return jobDeferred.promise;
    },config)
    .then(function(lyrics){
        console.log('Finished processing all the data');
        console.log(lyrics);
    });

To run the example in this repo run npm install and then node ./example/example.js

Please note this is a firing squad, you won't always have N functions executing at the same time, each function will wait for ALL OF THE OTHER functions of the squad to be finished before starting another batch.

That's all folks!