Feature request: add the ability to run "init" code at worker startup about node-worker-nodes

[kwiatkk1]

I think that I came up with the slightly better idea of solving this issue:

• we can take an advantage of that environment variables set in a master process are visible to its child processes (see: https://nodejs.org/api/process.html#process_process_env)
• so instead of hitting the filesystem and creating tmp files, we can simply use env to pass config data

So the code should look like this:

• in a master process, before spawning workers, create an env variable:

const WorkerNodes = require('worker-nodes');

async function run() {

    // setup env variable
    process.env.WORKER_OPTIONS = JSON.stringify({
        spam: 'eggs',
        ham: 'cheese',
        cacheSize: 100
    });

    // start processes...
    const workers = new WorkerNodes(require.resolve('./worker-source.js'), {
        autoStart: true,
        lazyStart: false,
        minWorkers: 4,
        maxWorkers: 4,
        maxTasksPerWorker: 1
    });

    // ...and wait for all the workers to be ready before proceeding
    await workers.ready();
}

run().then(() => console.log('ready'));

• in a worker's code (worker-source.js) read the env variable:

// receive config
const workerCustomOptions = JSON.parse(process.env.WORKER_OPTIONS);

console.log(`worker ${process.pid} up! custom options: ${JSON.stringify(workerCustomOptions)}`);

[csilvers]

Doh! -- a great approach. This is, like, what the environment is for...

[kwiatkk1]

Hi! Unfortunately there is no elegant solution for now to do this.

I think that it would be an useful option to pass some data on start bas you suggested. But for now, as workers don't receive any custom options during initialization phase, we can take advantage of how workers loads (they simply report ready if the source is loaded), and make some workaround:

• in a master process, before spawning workers, create a config file:

const WorkerNodes = require('worker-nodes');
const fs = require('fs');
const os = require('os');
const path = require('path');
const util = require('util');
const writeFileAsync = util.promisify(fs.writeFile);

async function run() {
    const tmpFilePath = path.join(os.tmpdir(), 'worker-startup-params.json');
    const workerInitOptions = {
        foo: 'bar',
        cacheSize: 100
    };

    // save worker config (hopefully using fast tmpfs access)
    await writeFileAsync(tmpFilePath, JSON.stringify(workerInitOptions));

    // start
    const workers = new WorkerNodes(require.resolve('./worker-source.js'), {
        autoStart: true,
        lazyStart: false,
        minWorkers: 4,
        maxWorkers: 4,
        maxTasksPerWorker: 1
    });

    // wait for all workers to be ready
    await workers.ready();
}

run().then(() => console.log('ready'));

• in a worker's code (worker-source.js) read the config:

const os = require('os');
const path = require('path');

const workerCustomOptions = require(path.join(os.tmpdir(), 'worker-startup-params.json'));

console.log(`worker ${process.pid} up! custom options: ${JSON.stringify(workerCustomOptions)}`);

This should produce output similar to the following:

> node index.js
worker 57621 up! custom options: {"foo":"bar","cacheSize":100}
worker 57622 up! custom options: {"foo":"bar","cacheSize":100}
worker 57624 up! custom options: {"foo":"bar","cacheSize":100}
worker 57623 up! custom options: {"foo":"bar","cacheSize":100}
ready

I hope it helps in your use case.

But if passing a config via tmp file is not an acceptable solution, please wait for a final one (a new option like initData):

const workers = new WorkerNodes(require.resolve('./worker-source.js'), {
    [...]
    initData: {
        cacheSize: 100
    }
});

[csilvers]

Thanks for the thoughtful reply! I ended up making a temporary file, just like you did, but I did it for the entire entrypoint, not just for the config reading, which ended up being simpler. My code is like this:

    const template = `${__dirname}/render_cache_${cacheSize}_XXXXXX.js`;
    const tmpobj = tmp.fileSync({template: template, discardDescriptor: true});
    const filename = tmpobj.name;
    const contents = `
const cache = require("./cache.js");
cache.init(${cacheSize});
module.exports = require("./render.js");
`;
    fs.writeFileSync(filename, contents);

    return new workerNodes(filename, options);

[spencermountain]

thanks craig and Krzysztof, if anyone is still reading this, I was able to pass a function into the worker using jsonfn.

This way you can execute arbitrary stuff, whenever you'd like.
cheers