Processor in separate process about bull

[fwoelffel]

I've had a quick look at this, and we can make this work 🎉

Here a few lines of code that depict this feature in action.

// app.module.ts
import { Module, OnModuleInit } from '@nestjs/common';
import { AppController } from './app.controller';
import { BullModule } from '../../lib';
import { join } from 'path';

@Module({
  imports: [
    BullModule.forRoot({
      processors: [ join(__dirname, 'processor.ts') ]
    }) 
  ],
  controllers: [ AppController ] // I add jobs through this controller
})
export class AppModule implements OnModuleInit {
  onModuleInit() {
    console.log('MAIN ', process.pid);
  }
}

// processor.ts
import {Job, DoneCallback} from 'bull';

export default function(job: Job, cb: DoneCallback) {
  console.log('CHILD ', process.pid);
  cb(null, 'It works');
}

Log output:

[Nest] 15191   - 05/07/2019, 10:23 PM   [NestFactory] Starting Nest application...
[Nest] 15191   - 05/07/2019, 10:23 PM   [InstanceLoader] BullModule dependencies initialized +17ms
[Nest] 15191   - 05/07/2019, 10:23 PM   [InstanceLoader] AppModule dependencies initialized +0ms
[Nest] 15191   - 05/07/2019, 10:23 PM   [RoutesResolver] AppController {/}: +4ms
[Nest] 15191   - 05/07/2019, 10:23 PM   [RouterExplorer] Mapped {/, GET} route +0ms
MAIN  15191
[Nest] 15191   - 05/07/2019, 10:23 PM   [NestApplication] Nest application successfully started +3ms
[ ... ] // Then I add a job to the queue
CHILD  15222

As you can see, the PID is not the same for the Nest app and the processor function.

[cgarnier]

I made a example project to make it explain my needs easier :D
https://github.com/cgarnier/nestjs-bull-example

I use pm2 to manage with the processes. The ecosystem file describes what it will start. 1 api and 4 workers.

There is 2 entrypoint, src/main.ts for the api, src/worker.ts for the workers. They boostrap respectivly AppModule and WorkderModule.
WorkerModule will imports only usefull stuff for the jobs and wont listen any ports.

Actulaly it s working. Jobs are load balanced between the processes and i can use the services into the workers processes.
But i cant avoid the api to run the jobs. I would like an option to disable the queue processing. processors: [] could do the trick (actualy it doesnt).

[fwoelffel]

Hi @iveselin

This has not been implemented but I imagine this is not a big deal. However, I don't think you would be able to use any service in your processor if it is executed in a separate process.
Feel free to have a look on this an submit a PR.

[klerick]

@fwoelffel PR #50 could add it. If create separate nest application with process decorator.

[cgarnier]

Hi @iveselin
I m looking for this feature too. Did you find a way to do it properly ?
I think it s possible to copy main.ts to worker.ts and remove app.listen. The worker should boostrap and the services should be available. But i dont know if it s possible to remove the useless of the app (controllers ) of the worker.
You can also do something similar with node cluster.

[fwoelffel]

I've done something similar not so long ago. I used a cluster in which the master created an app with the "normal" AppModule and the workers used a "lightweight" version of the AppModule (without controlers, etc.). What @cgarnier is describing could work perfectly but it wouldn't make use of the Bull feature to which @iveselin refers.
Maybe that's still an acceptable answer.

I'll also try to use the Bull feature, and document its usage with this module (I just have to find some time for this).

[cgarnier]

If you check the bull code, It s not 'real' processes anyway. They run the jobs in process.nextTick.

[fwoelffel]

Looks like an actual fork to me (https://github.com/OptimalBits/bull/blob/develop/lib/process/child-pool.js#L49). The easiest way to be sure would be to schedule a job writing its PID somewhere.

[fwoelffel]

Here is the currently possible types for the processors property:

export type BullQueueProcessor =
  | BullQueueProcessorCallback
  | BullQueueAdvancedProcessor;

export type BullQueueProcessorCallback = (
  job: Job,
  done?: DoneCallback,
) => void;

export interface BullQueueAdvancedProcessor {
  concurrency?: number;
  name?: string;
  callback: BullQueueProcessorCallback;
}

Here's my proposal:

export type BullQueueProcessor =
  | BullQueueProcessorCallback
  | BullQueueAdvancedProcessor
  | BullQueueSeparateProcessor
  | BullQueueAdvancedSeparateProcessor;

export type BullQueueSeparateProcessor = string;

export type BullQueueProcessorCallback = (
  job: Job,
  done?: DoneCallback,
) => void;

export interface BullQueueAdvancedProcessor {
  concurrency?: number;
  name?: string;
  callback: BullQueueProcessorCallback;
}

export interface BullQueueAdvancedSeparateProcessor {
  concurrency?: number;
  name?: string;
  processorPath: BullQueueSeparateProcessor;
}

@iveselin @cgarnier What do you think of this? Would this implementation bring an answer to your needs?

[iveselin]

@fwoelffel
Looks like an ok solution, little bit of confusion is when you use just a e.g. fuction name its executed in same process, but when you use a path to the file with the same function its executed as separate process... maybe add optional param in interface, something like separateProcess boolean?

Also, I see its a fork process, are services/repos accesable from that fork?

[fwoelffel]

As you've seen in Bull docs, passing a filesystem path is the way to handle jobs in a separate process. Moreover, that's the only way to start a fork. Given that the job processor lives in another process, you can't benefit from Nestjs' dependency injection, as said in my first answer.

If you want to handle jobs in separate processes and use services/repos, I suggest you to have a look at Node.js clusters.

[fwoelffel]

@iveselin @cgarnier feel free to review #74

I still have some to refactor some functions and add tests but the feature is implemented and working. Today is a day off, I'll try to merge before the end of this week.

[cgarnier]

I think we should not have to think about how are run the job in the app. I think it nice to have the possibility to start a number of workers process and let them pick the jobs. Maybe just some config to enable/disable the job execution by the main process and know who is a worker or not.
We can manage to start processes with cluster or pm2.

[cgarnier]

@fwoelffel is it possible to use the services in the pr example ?

[fwoelffel]

Maybe I'm misunderstanding what you mean but as I stated, services and any other components won't be available since the processor function will be running in a separate process. If you have an idea on how to get access to the Nest app components, you can provide a example or some instructions.

[fwoelffel]

@cgarnier I think this is out of the scope of this issue. Quick reminder: the point is to expose Bull's ability to run a processor in a nodejs fork (see https://github.com/OptimalBits/bull#separate-processes). Could you open a new issue if you need more help on the subject of jobs clustering and load balancing?

Anyway. Here's a few considerations about you're example:

• It's ok to do so;
• If your API is processing the jobs, it's because you provide it with MyQueue which handles twice and thrice jobs;
• If you're familiar with Docker, I'd advise you to containerize your app and set up replicas if you want to load balance your jobs ;)

[cgarnier]

Thanks, it s working.