learnboost / cluster

Node.JS multi-core server manager with plugins support.

Cluster

Cluster is an extensible multi-core server manager for node.js.

Installation

$ npm install cluster

Features

• zero-downtime restart
• hard shutdown support
• graceful shutdown support
• resuscitates workers
• maintains worker count, even if worker was _SIGKILL_ed.
• workers commit suicide when master dies
• spawns one worker per cpu (by default)
• extensible via plugins
• bundled plugins
  • cli: provides a command-line interface for your cluster
  • debug: verbose debugging information
  • logger: master / worker logs
  • pidfiles: writes master / worker pidfiles
  • reload: reloads workers when files change
  • repl: perform real-time administration
  • stats: adds real-time statistics to the repl plugin
• supports node 0.2.x
• supports node 0.4.x
• supports TCP servers

Example

app.js:

var http = require('http');

module.exports = http.createServer(function(req, res){
  console.log('%s %s', req.method, req.url);
  var body = 'Hello World';
  res.writeHead(200, { 'Content-Length': body.length });
  res.end(body);
});

server.js:

var cluster = require('cluster')
  , app = require('./app');

cluster(app)
  .use(cluster.logger('logs'))
  .use(cluster.stats())
  .use(cluster.pidfiles('pids'))
  .use(cluster.cli())
  .use(cluster.repl(8888))
  .listen(3000);

Note that cluster does not create these directories for you, so you may want to:

$ mkdir {logs,pids}

recommended usage: passing the path to prevent unnecessary database connections in the master process, as ./app is only require()ed within the workers.

var cluster = require('cluster');

cluster('./app')
  .use(cluster.logger('logs'))
  .use(cluster.stats())
  .use(cluster.pidfiles('pids'))
  .use(cluster.cli())
  .use(cluster.repl(8888))
  .listen(3000);

Plugins

Below are the known 3rd-party plugins for cluster:

• cluster-log remote logger powered by redis
• cluster-mail email exception notifications
• cluster-exception extensive exception notifications
• cluster-responsetimes response time statistics for cluster's REPL
• cluster-vhost add virtual host support to cluster

Screencasts

• Cluster Introduction

Running Tests

Install development dependencies:

$ npm install

Then:

$ make test

Actively tested with node:

• 0.4.9

Authors

• TJ Holowaychuk

License

(The MIT License)

Copyright (c) 2011 LearnBoost <[email protected]>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

cluster's Issues

Demote usr/grp

Restart and reload clears env, changing require.paths

I'm finding that when my app is restarted, either via the cli or automatically by reload, require.paths is reduced to:
[ '/usr/local/lib/node' ]
down from:
[ '/Users/sam/.npm',
'/Users/sam/.node_modules',
'/Users/sam/.node_libraries',
'/usr/local/lib/node' ]
causing "Cannot find module" errors.

I'm guessing this has something to do with process.ENV being replaced with:
{ CLUSTER_REPLACEMENT_MASTER: '1',
CLUSTER_PARENT_PID: '590',
__CF_USER_TEXT_ENCODING: '0x1F5:0:0' }

Mkdirs

for example when a user specifies a dir for log output, it may not exist, implement mkdir -p

Have workers re-spawn master if it dies

tough call here, hard to rely on node for things like this, maybe just best to leave monit in the picture but for those who have small sites this would be a nice feature, or if node can actually become stable enough to rely on the workers :)

Add dead worker stats

.

Add restart stats

.

Add some more tests

Spiky EINVAL

The simplest server test.js:

var cluster, http, server;
cluster = require('cluster');
http = require('http');
server = http.createServer(function(req, res) {
  var body;
  console.log('%s %s', req.method, req.url);
  body = 'Hello World';
  res.writeHead(200, {
    'Content-Length': body.length
  });
  return res.end(body);
});
cluster(server).set('workers',4).use(cluster.debug()).use(cluster.reload(['test.js'])).use(cluster.stats()).use(cluster.repl(30000)).listen(3000);

Below is typical debug output when starting workers. What can be the reason?

info �[90m- master started�[0m
info �[90m- worker 0 spawned�[0m
info �[90m- worker 1 spawned�[0m
info �[90m- worker 2 spawned�[0m
info �[90m- worker 3 spawned�[0m
info �[90m- worker 0 connected�[0m
info �[90m- worker 3 connected�[0m
Error: EINVAL, Invalid argument
at IOWatcher.callback (net.js:878:24)
�[31merror�[0m �[90m- worker 0 uncaught exception EINVAL, Invalid argument�[0m
Error: EINVAL, Invalid argument
at IOWatcher.callback (net.js:878:24)
�[31merror�[0m �[90m- worker 3 uncaught exception EINVAL, Invalid argument�[0m
�[33mwarning�[0m �[90m- worker 0 died�[0m
info �[90m- worker 0 spawned�[0m
�[33mwarning�[0m �[90m- worker 3 died�[0m
info �[90m- worker 3 spawned�[0m
info �[90m- worker 1 connected�[0m
info �[90m- worker 2 connected�[0m
info �[90m- listening for connections�[0m
Error: EINVAL, Invalid argument
at IOWatcher.callback (net.js:878:24)
�[31merror�[0m �[90m- worker 2 uncaught exception EINVAL, Invalid argument�[0m
Error: EINVAL, Invalid argument
at IOWatcher.callback (net.js:878:24)
�[31merror�[0m �[90m- worker 1 uncaught exception EINVAL, Invalid argument�[0m
�[33mwarning�[0m �[90m- worker 2 died�[0m
info �[90m- worker 2 spawned�[0m
�[33mwarning�[0m �[90m- worker 1 died�[0m
info �[90m- worker 1 spawned�[0m
info �[90m- worker 3 connected�[0m
info �[90m- worker 0 connected�[0m
info �[90m- worker 1 connected�[0m
info �[90m- worker 2 connected�[0m
info �[90m- shutting down�[0m
�[33mwarning�[0m �[90m- kill(SIGKILL)�[0m
info �[90m- shutdown complete�[0m
�[33mwarning�[0m �[90m- worker 3 died�[0m
�[90mdebug - exit�[0m

Add cli plugin

 $ node app.js status

etc

Previous processes are not killed after restart

When starting a cluster in the background (nohup & ...) and then using the cli plugin to restart the cluster, old master and workers processes aren not killed after the new main process has taken over. This can be reproduced with examples/cli.js as follows:

$ nohup node cli.js &
$ node test.js restart
debug - exit
$ node test.js restart
debug - exit
[1]+ Done nohup node test.js
$ node test.js restart
User defined signal 2
$ node test.js restart
User defined signal 2

In the meantime the amount of processes has already doubled/tripled... all the time no (persistent/browser) connections have been made - the above was just executed quickly one command after another.

spawn(-x)

Feasible to make spawn honor negative numbers to decrease the number of workers?

--Vladimir++

Support bind to domain name

cluster(server)
  .listen(80, 'example.com');

Add 0.2.x support

we would loose some cool functionality with this ... BLAH

Ignore node_modules when people reload(__dirname)

adds a LOT of stat watchers

Add .in('worker', callback)

does not output console.log()

I am using latest Express with cluster, my node version is v0.4
var cluster = require('cluster');
var express = require('express');
var app = module.exports = express.createServer();

    app.get('/', function(req, res){

      console.log('New Request'); // this does not output

      res.render('index', { title: 'Express' });
    });

    cluster(app)
        .use(cluster.logger('logs'))
        .use(cluster.stats())
        .use(cluster.pidfiles('pids'))
        .use(cluster.cli())
        .use(cluster.repl(8888))
        .use(cluster.reload(['app.js']))
        .use(cluster.debug())
        .listen(3000);

It doestn't output the message using console.log() in the console or master.log or workers.access.log.
I am not sure it is a bug or by design?
Thanks

Have reload() filter out non-js files

currently cluster (by default) writes to some files in the script's directory, causing a cyclic restart. Template engines + css etc already can be modified on the fly so we should only really watch for *.js files

Report exceptions to master

I haven't started using cluster yet, but I saw this in worker.js:

process.on('uncaughtException', function(err){
    console.error(err.stack || err.message);
    self.destroy();
});

This should be optional because (if I'm not wrong... oh, asynchronous stuff), self.destroy() might be called before exceptional-node's sending call.

Add diagrams to explain plugin / IPC architecture etc

Fix stats for 0.2.x due to os module

os is currently required for some stats

Add repl support to cli()

as per guillermo's request :) this is a good idea though, telnet seems to suck on our server

Add docs for defining repl functions

-s both for status and shutdown

in cli plugin, "-s" is used for both status and shutdown commands

intercommunication channel

Hi!

Please, consider adding a duplex way for workers to broadcast simple JSON messages. It may be achieved via a shared unix socket, like it's done at https://github.com/dvv/simple/blob/master/src/server.coffee#L194 for master and at https://github.com/dvv/simple/blob/master/src/server.coffee#L77 for workers.

I just discovered cluster and want to switch to it and to cease maintaining own solution, but I need intercommunication so that workers can report to each other changes in their state. Having redis just for pubsub seems overhead. Another pubsub solutions are either immature or too complex to employ.

TIA,
--Vladimir++

Error: ENOTSUP, Operation not supported (_doListen)

Hi, I want to use http://learnboost.github.com/cluster/ on my server, but with the demo code on github, I get this error message:

events.js:23
throw arguments[1]; // Unhandled 'error' event
                   ^
Error: ENOTSUP, Operation not supported
at Server._doListen (net.js:1062:5)
at net.js:1010:16

NodeJS version is v0.4.2.

app.js:

var http = require('http');
module.exports = http.createServer(function(req, res){
  console.log('%s %s', req.method, req.url);
  var body = 'Hello World';
  res.writeHead(200, { 'Content-Length': body.length });
  res.end(body);
});

server.js:

var cluster = require('cluster')
, app = require('./app');
cluster(app)
  .use(cluster.logger('logs'))
  .use(cluster.stats())
  .use(cluster.pidfiles('pids'))
  .use(cluster.cli())
  .use(cluster.repl(8888))
.listen(3000);

Can anybody help?

Indicate worker state in stats

it would be nice to list the workers and their current state, aka active, shutting down etc

Allow stats() prev master

currently the new master kicks in on restart, clobbers the old one in terms of the REPL, so either we change the path for the old one so you can still check the status, or via IPC grab stats from the previous master

Add light-weight supervisor process

as a plugin, but we need 0.4.1 for the null signal patch

Prevent restart when restarting

lots of procs :)

Add log rotation capabilities

logger() exposes the streams, so we could add logrotate()

Better async startup / shutdown hooks

currently the events fire, but they do not allow the shutdown to halt until cleanup is complete, removing files, sockets etc

something is totally broken

doesn't matter what I trying to do, I get always this

events.js:23
        throw arguments[1]; // Unhandled 'error' event
                       ^
Error: EINVAL, Invalid argument
    at Server._doListen (net.js:1062:5)
    at net.js:1010:16

now using node v0.4.0, on macos.

I have tried your examples and to run tests, nothing worked, always with the same error.

Fix cli() status command

waiting on this node patch: https://github.com/ry/node/pull/674/files

Add conditional plugins based on NODE_ENV

example:

cluster(server)
  .set('workers', 4)
  .in('development').use(cluster.logger('logs', 'debug'))
  .in('development').use(cluster.debug())
  .in('production').use(cluster.pidfiles())
  .listen(3000);

Add notification system plugin (hoptoad etc)

could be third-party

cannot find module 'log'

node version v0.3.7, macos

node.js:116
throw e; // process.nextTick error, or 'error' event on first tick
^
Error: Cannot find module 'log'
at Function._resolveFilename (module.js:204:11)
at Function._load (module.js:162:25)
at require (module.js:231:19)
at Object. (/usr/local/lib/node/.npm/cluster/0.0.2/package/lib/plugins/logger.js:13:11)
at Module._compile (module.js:287:26)
at Object..js (module.js:293:10)
at Module.load (module.js:219:31)
at Function._load (module.js:186:10)
at require (module.js:231:19)
at Function.logger (/usr/local/lib/node/.npm/cluster/0.0.2/package/lib/cluster.js:46:12)

os.cpus() undefined

I'm using the simple example app.js you give; server.js is just
var cluster = require('cluster')
cluster('./app').listen(8001)

I get the following error:

$ node server.js 

node.js:116
        throw e; // process.nextTick error, or 'error' event on first tick
        ^
TypeError: Cannot read property 'length' of undefined
    at Master.start (/usr/local/lib/node/.npm/cluster/0.3.2/package/lib/master.js:393:18)
    at /usr/local/lib/node/.npm/cluster/0.3.2/package/lib/master.js:253:16
    at Master.createSocket (/usr/local/lib/node/.npm/cluster/0.3.2/package/lib/master.js:366:5)
    at Master.listen (/usr/local/lib/node/.npm/cluster/0.3.2/package/lib/master.js:250:14)
    at Object.<anonymous> (/usr/home/elijah/cluster/server.js:3:18)
    at Module._compile (module.js:373:26)
    at Object..js (module.js:379:10)
    at Module.load (module.js:305:31)
    at Function._load (module.js:271:10)
    at Array.<anonymous> (module.js:392:10)

It looks like os.cpus() is undefined in
if (!this.has('workers')) {
this.set('workers', os
? os.cpus().length
: 1);
}

Add email notification plugin

could be third-party

Restart not working as expected.

Seeing some unexpected behavior when attempting to restart master / workers programatically. I've summarized the problem here: https://gist.github.com/e4ef8fcbb5be2f468967

Is this an issue or am I doing something wrong?

Add option to bind/listen when the first worker is connected

currently all n workers must be connected

Maintain worker count

currently if you SIGKILL a worker, master is not reported, and ha no way of knowing it lost a worker. Again we need the null signal support in the upcoming node 0.4.1 to allow the worker count to be maintained, until then, we have to rely on the workers contacting master when they die

Add cluster(filename) support

allowing cluster to require the exported server only when in a worker to prevent these issues with db connections being open in master etc

Add geckoboard-like http admin interface

http://www.geckoboard.com/

Add docs for socket write helpers

title() and row()

Allow plugins to act within workers

Currently they work only in the master process, so you cannot check server connections etc. it would be expensive to report such things back, but plugins should still be able to function at the worker level, enabling active connection stats etc

Not working under Cygwin

My Express application starts with no errors and spawns the workers but the site is not accessible when started under Cygwin.

On MacOS it works perfectly.

Add transparent https support

Add optional but transparent support for duplicating the server on 443 with the given options

Come up with better boot solution

master does not need to require everything that the user's server does, does not need to connect to databases etc, so this can cause master to hang when restarting etc. We have a few options... including supplying a path to cluster() to load the app but ONLY in the workers.

regardless we need docs for this

Attach node debugger

I created the following CLI command to attach the node debugger but it doesn't seem to be working and I'm not sure why.

cluster.cli.define('-d, --debug, debug', function(master) {
  master.kill('SIGUSR1');
  process.exit();
}, 'Attach the node debugger to workers');

I was under the impression that signalling the worker processes with SIGUSR1 should attach the debugger. The bottom of this page explains it: http://nodejs.org/docs/v0.4.1/api/debugger.html

What ends up happening is that all workers are restarted. I didn't think that master.kill('SIGUSR1') would do that. Is there another way to attach the debugger to workers?

What I would really love is for it also to work after workers reload from the 'reload' plugin. Possible?

EventEmitter

Hi,

I tried to run the simple example in Readme.md on my computer (Ubuntu 10.10, 64bits) with node 0.4.0, but I get the following error (appears multiple times) :

(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace:
at [object Object]. (events.js:101:17)
at Socket.pipe (stream.js:67:8)
at Master. (/home/tanguy/local/lib/node/.npm/cluster/0.0.3/package/lib/plugins/logger.js:136:21)
at Master.emit (events.js:59:20)
at Master.spawnWorker (/home/tanguy/local/lib/node/.npm/cluster/0.0.3/package/lib/master.js:310:8)
at Master.workerKilled (/home/tanguy/local/lib/node/.npm/cluster/0.0.3/package/lib/master.js:461:12)
at Master.invoke (/home/tanguy/local/lib/node/.npm/cluster/0.0.3/package/lib/mixins/receiver.js:51:18)
at Master.frame (/home/tanguy/local/lib/node/.npm/cluster/0.0.3/package/lib/mixins/receiver.js:32:14)
at Socket. (native)
at Socket.emit (events.js:42:17)

Here is an extract of master.log

[Wed, 16 Feb 2011 23:32:10 GMT] INFO master started
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 0
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 1
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 2
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 3
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 4
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 5
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 6
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 7
[Wed, 16 Feb 2011 23:32:10 GMT] ERROR worker 3 died
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 3
[Wed, 16 Feb 2011 23:32:10 GMT] ERROR worker 1 died
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 1
[Wed, 16 Feb 2011 23:32:10 GMT] ERROR worker 0 died
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 0
[Wed, 16 Feb 2011 23:32:10 GMT] ERROR worker 5 died
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 5
[Wed, 16 Feb 2011 23:32:10 GMT] ERROR worker 2 died
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 2
[Wed, 16 Feb 2011 23:32:10 GMT] ERROR worker 6 died
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 6
[Wed, 16 Feb 2011 23:32:10 GMT] ERROR worker 4 died
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 4
[Wed, 16 Feb 2011 23:32:10 GMT] ERROR worker 7 died
[Wed, 16 Feb 2011 23:32:10 GMT] INFO spawned worker 7
[Wed, 16 Feb 2011 23:38:26 GMT] ERROR worker 7 died
[Wed, 16 Feb 2011 23:38:26 GMT] INFO spawned worker 7
[Wed, 16 Feb 2011 23:38:26 GMT] ERROR worker 6 died

Not sure if this is cluster-specific though...

Thanks in advance!