watson / worker-threads-pool

Easily manage a pool of Node.js Worker Threads

worker-threads-pool

Easily manage a pool of Node.js Worker Threads.

Installation

npm install worker-threads-pool --save

Prerequisites

Worker Threads in Node.js are still an experimental feature and is only supported in Node.js v10.5.0 and above. To use Worker Threads, you need to run node with the --experimental-worker flag:

node --experimental-worker app.js

Usage

const Pool = require('worker-threads-pool')

const pool = new Pool({max: 5})

for (let i = 0; i < 100; i++) {
  pool.acquire('/my/worker.js', function (err, worker) {
    if (err) throw err
    console.log(`started worker ${i} (pool size: ${pool.size})`)
    worker.on('exit', function () {
      console.log(`worker ${i} exited (pool size: ${pool.size})`)
    })
  })
}

API

pool = new Pool([options])

options is an optional object/dictionary with the any of the following properties:

• max - Maximum number of workers allowed in the pool. Other workers will be queued and started once there's room in the pool (default: 1)
• maxWaiting - Maximum number of workers waiting to be started when the pool is full. The callback to pool.acquire will be called with an error in case this limit is reached

pool.size

Number of active workers in the pool.

pool.acquire(filename[, options], callback)

The filename and options arguments are passed directly to new Worker(filename, options).

The callback argument will be called with the an optional error object and the worker once it's created.

pool.destroy([callback])

Calls worker.terminate() on all workers in the pool.

Will call the optional callback once all workers have terminated.

License

MIT

worker-threads-pool's Issues

pooL.aqcuire never finihsed

Hi,
First i wanted to say THANKS for a great ,simple to use component.
We are using it a lot and its seems like working fine for few good months UNTIL 2 days ago.
We noticed a strange behavior lately from a very simple code:

CODE

var workerPool = require('worker-threads-pool');
var pool = new workerPool ({max: 5});
pool.acquire(destdir, {
.....
The strange part is the the pool never able to acquire a new resource and our code not callback.
Any chance that you will consider adding :
Timeout for acquiring ? so if after a specific time the pool is not acquiring it will throw an error ?
We've upgraded to v2 and we saw that there is a new maxWaiting but we are afraid that we will end up with the same problem as happened prior v2.

If needed, you can add my as a collaborator and ill help.
The reported issue is not happening all the time and we are not able to reproduce it.
Thanks!

Please publish 2.0

The latest release on npm is 1.0. Please publish the latest version. Thanks!

Reuse workers

Currently the package creates a new worker for each acquire and the pool is more of a cache for checking length and cleanup.

Are there any thoughts around looking to add an option to reuse workers via channel messages if they have the same filenames?

The question about the example?

Some question about the example? the example you give the tasks to run by thread pool, if the tasks number is dynamic, how to resolve this problem? The following code:

const Pool = require('worker-threads-pool')
const pool = new Pool({max: 5})
const tasks=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17]
for (let i = 0; i < tasks.length; i++) {
pool.acquire('./worker.js',{workerData:tasks[i]}, function (err, worker) {
if (err) throw err
console.log(started worker ${i} (pool size: ${pool.size}))
worker.on('exit', function () {
console.log(worker ${i} exited (pool size: ${pool.size}))
})
})
}
console.log("main thread")

thanks!

Where is the module 'worker_threads'?