ejhayes / nestjs-bullmq

DEPRECATED

This is now built into NestJS as of nestjs/bull#1129. Please consider switching to the officially supported version instead of this package.

npm i --save @nestjs/bullmq

Description

BullMQ module for Nest. This should hopefully be a drop-in replacement for @nest/bull package.

There is one caveat with this implementation: Queue Schedulers. This library does not automatically create a queue scheduler because of the potential drain on resources:

It is ok to have as many QueueScheduler instances as you want, just keep in mind that every instance will perform some bookkeeping so it may create some noticeable CPU and IO usage in your Redis instances.

PRs welcome to fix this!

Installation

$ npm i --save @ejhayes/nestjs-bullmq

Quick Start

Overview & Tutorial

License

Nest is MIT licensed.

nestjs-bullmq's Issues

Named processors

I'm submitting a...

[x] Regression 
[ ] Bug report
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead post your question on Stack Overflow.

Current behavior

If using this library and you have:

@Process({
  name: 'NAMED_JOB_1',
  concurrency: 5,
})
async processSomeJob1(job: Job<any>): Promise<void> {
  // logic here
}

@Process({
  name: 'NAMED_JOB_2',
  concurrency: 5,
})
async processSomeJob2(job: Job<any>): Promise<void> {
  // logic here
}

Then both functions will get called regardless of the job name passed in.

Expected behavior

Specifying a named job should result in the correct processor being used. This functionality currently exists with the nestjs/bull package.

Minimal reproduction of the problem with instructions

The following should work:

@Process({
  name: 'A_NAMED_JOB',
  concurrency: 5,
})
async processSomeJob(job: Job<any>): Promise<void> {
  // logic here
}

What is the motivation / use case for changing the behavior?

Making this work would help ensure that this library remains a drop in replacement for the nestjs/bull package.
taskforcesh/bullmq#448 (comment)

Functionality wise, Bullv3 and BullMQ provides the same, it just that the syntax is different. In BullMQ you only define 1 processor, and then inside the processor you can choose different code paths depending on the "job.name" property. In your case this will mean a bit more of code, but named processors where causing a lot of confusion so we decided to remove the syntax sugar and made it more explicit.

Environment

For Tooling issues:
- Node version: 12
- Platform:  Mac

Proposal for QueueScheduler handling

See

nestjs-bullmq/lib/bull.providers.ts

Lines 23 to 33 in 683673e

	/**
	* TODO: Add top level option to allow for creation of scheduler. This current
	* implementation currently would cause multiple schedulers to be created if
	* we were scaling horizontally.
	* @url https://docs.bullmq.io/guide/queuescheduler
	*/
	/*
	if (!option.disableScheduler) {
	workers.push(new QueueScheduler(queue.name, option))
	}
	*/

There is an open issue (taskforcesh/bullmq#1075) to add a method of checking for existing QueueSchedulers on BullMQ. Once that feature is added, we can use it to configure QueueSchedulers as a part of this module, instead of leaving it to the user.
The module config options would take in an option for a number of schedulers. On creation, we would wait for a random time ( to avoid multiple horizontally scaled apps from checking this at the exact same time), check for the number of current schedulers, and create a scheduler if less than the settings.

Some considerations:

• Concurrency issues if all replicas start together. I don't know if the random wait is a good approach
• Rather than a global setting, users may want to vary this per queue. Higher redundancy for more important queues for example
• The scheduler should be injectable in the module

The automated release is failing 🚨

🚨 The automated release from the master branch failed. 🚨

I recommend you give this issue a high priority, so other packages depending on you can benefit from your bug fixes and new features again.

You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. I’m sure you can fix this 💪.

Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.

Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the master branch. You can also manually restart the failed CI job that runs semantic-release.

If you are not sure how to resolve this, here are some links that can help you:

• Usage documentation
• Frequently Asked Questions
• Support channels

If those don’t help, or if this issue is reporting something you think isn’t right, you can always ask the humans behind semantic-release.

---

Missing package.json file.

A package.json file at the root of your project is required to release on npm.

Please follow the npm guideline to create a valid package.json file.

---

Good luck with your project ✨

Your semantic-release bot 📦🚀