quirrel-dev / quirrel Goto Github PK

Bug Report

Current Behavior

While navigating on ui.quirrel.dev I keep getting CORS errors while trying to connect on api.quirrel.dev.

Access to fetch at 'https://api.quirrel.dev/health' from origin 'https://ui.quirrel.dev' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Bug Report

Current Behavior
Queueing using a CRON using last day of month L symbol (0 0 10 L * *) it seems that this is taking the last day of the current month. Example queuing it today (03/31/2021) with 0 0 10 L * * In the UI I see the following CRON scheduled: 0 0 10 30 * *

Input Code

• REPL or Repo link if applicable:

queue.enqueue(
  {  ...  },
  {
    id: `xxx`,
    override: true,
    repeat: {
      cron: `0 0 10 L * *`
    }
  }
)

Expected behavior/code
I would expect to have a job executed last day of each month.

Environment

• Quirrel version: v1.3.3
• Node/npm version: v12.13.1

See #49 (comment):

@Kerumen: "I've just installed Quirrel latest version and I have the same problem. However, I have a 500 on the WS handshake"

At the moment, a job with a past runAt will be sent to the API and result in a cryptic error message. This should already throw on the application side.

Setting a delay doesn't actually delay the execution of the job. I've only tested this in development mode so not sure if it's also a problem in the hosted version? With the following enqueue the job gets executed immediately:

import reminderQueue from "./queues/reminder";

// pages/api/setupReminder.js
export default async (req, res) => {
  const email = req.body;
  await reminderQueue.enqueue(email, {
    id: email,
    delay: "1m",
    repeat: {
      every: "1m",
      times: 1,
    },
  });
  res.status(200).end();
};

Full repro here: https://github.com/maxfi/quirrel-dev-water-reminder-demo

When creating a bookmark in https://ui.quirrel.dev/ for an instance, a dialogue box comes up with the URL. However, this is not selectable for me. So not able to copy and paste to create the bookmark.

Fully client-side. Uses symmetric encryption. If no key is specified, we use the Token for our key.

Currently, if I save a CronJob file while it contains illegal syntax, the Quirrel CLI prints an error. This shouldn't happen.

Bug Report

Current Behavior
Quirrel API returns 404 error when trying to create or query a queue with endpoint URL longer than 86 characters long.

Input Code

// QUIRREL_BASE_URL=https://some-very-long-subdomain.example.org
const PATH = 'very/long/queue/endpoint/url/with/a/lot/of/characters'
const queue = Queue(PATH, () => {})
queue.enqueue(...)

Expected behavior/code
I expect the job to be created.

Environment

• Quirrel version: [e.g. v1.5.0]
• Node/npm version: [e.g. Node 12/npm 7]

Additional context/Screenshots
I think this should be a pretty common case when working with Vercel preview environments. Our preview environments' domain names are 50 just by themselves.

This is what you get when trying to create a job with a long endpoint URL:

And this is what you get with URL not that long:

Bug Report

Current Behavior
I tried to access the docs on Swagger but it's currently down: https://api.quirrel.dev/documentation

Also, what bearer token should I use to be able to make requests on Postman for example?

We don't seem to be receiving any messages back that we have queued up. I went to the site to try and log into the dashboard (https://quirrel.dev/) to see what was going on and its throwing a 500.

Some jobs, some of the time, are OK to fail and retry.
(that was a rhyme, nice!)

Quirrel should have some form of retry / backoff.

When I run the quirrel command to start the server, I get lots of errors similar to Error: EMFILE: too many open files

Error: EMFILE: too many open files, open 'widget/node_modules/wrap-ansi/node_modules/ansi-regex/index.js'
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
Error: EMFILE: too many open files, open 'widget/node_modules/wrap-ansi/node_modules/strip-ansi/index.d.ts'
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
Error: EMFILE: too many open files, open 'widget/node_modules/wrap-ansi/node_modules/strip-ansi/index.js'
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
Error: EMFILE: too many open files, open 'widget/node_modules/yargs-parser/node_modules/camelcase/index.d.ts'
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
Error: EMFILE: too many open files, open 'widget/node_modules/yargs-parser/node_modules/camelcase/index.js'
....

I am using WSL2 with redwood client libraries. I am running the command with yarn run quirrel

Feature Request

When Quirrel is integrated with Blitz or maybe this counts for Next.js as, then it will require to have the ENV QUIRREL_BASE_URL set, but at build time, that URL might not be known and can change later.

QUIRREL_BASE_URL could be removed for build time, so no jobs can be executed at build time.

(copied from #544:)

Now we have the inverse happening @Skn0tt

Every deployment, it's telling me they will be removed... every deployment. I did remove the code, but I'm not sure why it's happening every time. I'm also seeing in my logs that the cron is indeed still being run hourly.

Originally posted by @saiichihashimoto in #544 (comment)

Bug Report

Current Behavior
I have followed the steps to deploy Quirrel on my own hosting. but when I get to this step:
https://docs.quirrel.dev/deploying#acquire-a-token
I get an 404 when I try to do that request.
I can access ui and login so thats not an problem :)

Expected behavior/code
I can do https://docs.quirrel.dev/deploying#acquire-a-token

Environment

• Quirrel version: 1.6.1

Feature Request

For better readability, there should be a way to specify the timezone in which a cron job is interpreted.

// just an example
export default Cron("...", "* * * * 1", "Europe/Berlin", async () => ...)

Bug Report

Current Behavior
TimeoutOverflowWarning enqueuing in Blitz: TimeoutOverflowWarning: 2650483302 does not fit into a 32-bit signed integer.

Input Code

await reminderQueue.enqueue({
    to: "Sandy Cheeks",
    message: "Howdy!",
  }, {
    repeat: {
      cron: '0 10 18 * *'
    }
  })

Expected behavior/code
No warnings

Environment

• Quirrel version: v1.3.1
• Node/npm version: v12.13.1

Additional context/Screenshots

In order to save some space and allow duplicate usage of jobId, completed jobs should be deleted after execution.

The Headbar isn't responsive anymore, happened sometime in the last commits.

error [email protected]: The engine "node" is incompatible with this module. Expected version "12 || 14 || 15". Got "16.4.0"

Could we please have Node 16 support.

Bug Report

Current Behavior
I am running Redis and Quirrel in Kubernetes cluster. At the moment there are just two pods. After a while the token get's invalid. This might be the case, when Quirrel or Redis is restarted in the Kubernetes node.

Expected behavior/code
Generated Tokens should be persistent and saved in a stateful environment.

Environment
Kubernetes, Pods

• Quirrel version: ghcr.io/quirrel-dev/quirrel:sha-e27789b

Possible Solution

Maybe a persistent volume for Redis? So even if Redis, if Redis holds the Tokens, should re-read the data from persistent storage.

The imports are broken on v1.3.0. Rolled back to 1.2.0 and everything works as expected.

Bug Report

Current Behavior
Running quirrel ci as part of the Vercel build seems to cause chokidar to watch the entire directory and exceed the system limit for number of file watchers.

package.json:

{
  "verce-build": "quirrel ci"
}

Repo

Expected behavior/code
The build runs Quirrel CI with no issues.

Environment

• Quirrel version: v1.0.4
• Node/npm version: Node 14 / npm 7

Possible Solution

Unable to recreate locally. Originally thought it was because node_modules wasn't gitignored, but it occurs with it there. Vercel truncates the logs to 2000 lines so I'm not able to see the top of the stack trace (AFAIK?)

Please let me know if there's any more detail I can provide!

Additional context/Screenshots

The CLI prints about a detected cron job on every save, this shouldn't need to be.

GET /jobs should return all pending jobs.

I've recently asked this on Twitter (see https://twitter.com/sehmaschine/status/1350415050822066176).

We have different kinds of background tasks with our appliations:

1. Background tasks which are independent of a user action, e.g. sending daily mails or doing backups. These tasks are usually part of a distinct application/microservice. I can see how these tasks are handled with Quirrel.
2. Long running jobs which have to run in the background, e.g. generating PDFs. These tasks are part of the main app and are triggered by a user action (e.g. clicking a button). Then the user either stays on the page and needs updates on the process or the user leaves the page and we will inform her about the process later on. With this usecase, we need to update a single job with the current state (e.g. 23 of 80 PDFs created) regularly.

When looking at the Quirrel pricing page (which refers to API calls), it seems that usecase 2 has not been a major concern, right? Because in that case I would (for example) have 100 API calls for 1 process (because of the in-between process updates). I just wanna make sure we are not backing the wrong horse ...

None of the CronJobs are calling our endpoints when I deploy them on vercel and have vercel ci inside of my vercel-build script, and I can confirm it's actually building in the build log.

In the quirrel dashboard, the API Calls are going up, so it looks like it's actually doing something:

However, in vercel, I'm not seeing any logs in my log drain of the endpoint being called whatsoever. When I go to it in the browser (which obviously won't work) it DOES generate a log of visiting the endpoint, so I know if quirrel was actually reaching it then I would see something.

// src/pages/api/v1/crons/orders/cleanup/index.ts
import { CronJob } from 'quirrel/next';

const cron = CronJob('api/v1/crons/orders/cleanup', '0 * * * *', async (...args: any[]) => {
	...
});

export default cron;

It's not the file pathing. I have a Queue also running using the same file structure and path so, as long as it's consistent between them, then it should be fine.

How do I make this work?

Hello,

⚠️ We're getting errors with "examples/nuxt - Npm" and have given up after trying several times.

In most cases that means something is wrong with your current Npm setup and we can't fix it automatically:

• Error details:

Read-only file system @ dir_s_mkdir - /tmp/d20210209-4-xn40qm/../../quirrel

• Error details:

Read-only file system @ dir_s_mkdir - /tmp/d20210209-4-11iz6ea/../../quirrel

• Error details:

Read-only file system @ dir_s_mkdir - /tmp/d20210209-4-1elkpsk/../../quirrel

After you've fixed the problem, please activate this project again in the Depfu Dashboard.

👉 We will not send you further PRs until this is fixed and the repo is activated again.

If you need help or this looks like an error on our side, please send us an email.

Is there an update on this? quirrel-dev/quirrel-next#3

This has happened a few times when I leave the server running.

Bug Report

Current Behavior
Leaving Quirrel dev server running for several minutes

Input Code
concurrently 'next dev' 'quirrel'

Expected behavior/code
Server keeps running.

Environment

• Quirrel version: 1.5.0
• Node/npm version: Node 15.12.0/NPM 7.9.0

Additional context/Screenshots
[1] This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
[1] TypeError: Cannot read property 'split' of undefined
[1] at StaleChecker.parseJobDescriptor (/Users/bsolvang/Documents/SandeFredin/clients/klinikk.no/node_modules/@quirrel/owl/dist/shared/stale-checker.js:98:29)
[1] at Array.map ()
[1] at StaleChecker. (/Users/bsolvang/Documents/SandeFredin/clients/klinikk.no/node_modules/@quirrel/owl/dist/shared/stale-checker.js:126:117)
[1] at step (/Users/bsolvang/Documents/SandeFredin/clients/klinikk.no/node_modules/@quirrel/owl/dist/shared/stale-checker.js:33:23)
[1] at Object.next (/Users/bsolvang/Documents/SandeFredin/clients/klinikk.no/node_modules/@quirrel/owl/dist/shared/stale-checker.js:14:53)
[1] at fulfilled (/Users/bsolvang/Documents/SandeFredin/clients/klinikk.no/node_modules/@quirrel/owl/dist/shared/stale-checker.js:5:58)
[1] at runNextTicks (node:internal/process/task_queues:59:5)
[1] at listOnTimeout (node:internal/timers:526:9)
[1] at processTimers (node:internal/timers:500:7)

I've started the development server using concurrently as the steps were mentioned in the documentation.

But I'm getting this error and the dev server for quirrel keeps refreshing due to which a cron-job keeps getting added to the list.

pages/api/cron.ts

import fetch from "node-fetch";

export default CronJob("api/delete", "0 * * * *", async (job) => {
	try {
		console.log(job);
		const response = await fetch("/api/delete");
		const data = await response.json();

		console.log("data ---> ", data);
	} catch (error) {
		console.error(error);
	}
});

TypeError: connection.socket.send is not a function
    at /Users/abhishekkumar/Desktop/qrshare/node_modules/quirrel/dist/src/api/scheduler/routes/activity.js:68:43
    at JobsRepo.<anonymous> (/Users/abhishekkumar/Desktop/qrshare/node_modules/quirrel/dist/src/api/scheduler/jobs-repo.js:257:21)
    at step (/Users/abhishekkumar/Desktop/qrshare/node_modules/quirrel/dist/src/api/scheduler/jobs-repo.js:63:23)
    at Object.next (/Users/abhishekkumar/Desktop/qrshare/node_modules/quirrel/dist/src/api/scheduler/jobs-repo.js:44:53)
    at /Users/abhishekkumar/Desktop/qrshare/node_modules/quirrel/dist/src/api/scheduler/jobs-repo.js:38:71
    at new Promise (<anonymous>)
    at __awaiter (/Users/abhishekkumar/Desktop/qrshare/node_modules/quirrel/dist/src/api/scheduler/jobs-repo.js:34:12)
    at Activity.onEvent (/Users/abhishekkumar/Desktop/qrshare/node_modules/quirrel/dist/src/api/scheduler/jobs-repo.js:253:74)
    at Activity.<anonymous> (/Users/abhishekkumar/Desktop/qrshare/node_modules/@quirrel/owl/dist/activity/activity.js:105:51)
    at step (/Users/abhishekkumar/Desktop/qrshare/node_modules/@quirrel/owl/dist/activity/activity.js:33:23)```

As some users may know, I've been talking about the issues in figuring out decent Cron job support
basically since the inception of Quirrel.

Over past weeks, I got to present these issues to a lot of you, receiving valuable feedback.
You had great questions & ideas, and over time all of those intermingled to form the following RFC.

How Cron jobs look like today:

Quirrel already supports something it calls "cron jobs":

billingQueue.enqueue(
    undefined,
    {
        id: "billing", // makes sure job isn't duplicated
        override: true, // if another job with that ID already exists, override it
        repeat: {
            cron: "* * * * 1 *"
        }
    }
)

There are three main problems with this:

1. It is non-obvious that Cron jobs are created using Queues (outside of Quirrel, these are two distinct concepts)
2. It requires developers to remember calling /api/registerCronJobs route whenever they changed their CRON schedules, which is clunky at best.
3. It separates the Cron handler and the (highly cohesive) Cron schedule into two files.

Let's see if we can do better 🤔

How Cron jobs should look like instead:

Setting up a new CronJob should be as easy as importing CronJob:

// pages/api/billingCron
import { CronJob } from "@quirrel/next"

export default CronJob(
  "billingCron",
  "5 4 * * 1", // at 04:05 on monday
  async () => {
    // do something
  }
);

Registering that CRON job with the Quirrel API can then be done by running quirrel register-cron anywhere in your application directory.
When run during development, this would register it against the development version.

Running quirrel register-cron --prod can then be run during build time to register against the production Quirrel instance.

This solves all of the above-mentioned problems:

1. It adds a new, distinct concept for creating Cron Jobs
2. It still requires you to call quirrel register-cron, but that can easily be integrated automated into your build process
3. It keeps the Cron schedule in the same file as the corresponding handler.

Okay, now that we know what we're aiming for, we can think about how to build this!

How the improved could be implemented

There's two main things we need to work out here:

1. How can quirrel register-cron detect all my Cron jobs?
2. What happens to Cron jobs when they're no longer needed?

How Cron jobs can be detected

Short answer: through static analysis (aka. "parsing your code").

Basically, we need to look for files containing two things:

1. import { CronJob, ... } from "@quirrel/next"
2. CronJob(<queueName>, <cronSchedule>, async () => { ... })

We can then use the extracted queueName and cronSchedule to make the respective calls to the Quirrel API.

In the beginning, extraction can be performed using simple RegEx-based parsing.
This should cover most (idiomatic) uses but if the need arises in the future, we could just go straight-overkill and use Babel.

One thing to consider: upcoming client libraries for frameworks other than Next.js should easily be made compatible with this.
The logic of detecting Cron jobs should therefore be deferred to the installed client library.

What happens to Cron jobs when they're no longer needed?

I've been thinking about this question for a long time, and it's why this draft took so long!
I'm only gonna go into the solution here, but feel free to hit me up for hearing the problems I found with other approaches.

Cron jobs can become "no longer needed" when:

• their deployment is torn down (e.g. Vercel's preview deployments)
• the cron job handler is removed

Both of these can be detected by examining the HTTP statuses:
If a job's execution response is a 404, its queue is no longer needed.

So what do we do about that queue?
Now that we have incident forwarding, that question is easy to answer:
We discard all upcoming jobs on that queue and put the data into a new incident.
This allows administrators to remedy the removal of those jobs in case it was actually caused by an infrastrucutre bug.

Conclusion

Congrats, you made it to the end!

What do you think about the proposed changes?
Do they make sense?
Do they really improve things?

I greatly appreciate comments below ☺️

Bug Report

I'm following the tutorial on Processing Uploads in the Background. When I try to install the quirrel/blitz package, I get an error--

yarn add quirrel/blitz && npm run dev
yarn add v1.22.10
[1/4] Resolving packages...
error Command failed.
Exit code: 128
Command: git
Arguments: ls-remote --tags --heads ssh://[email protected]/quirrel/blitz.git
Directory: /home/chris/Documents/giftalert
Output:
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

Input Code

yarn add quirrel/blitz

Expected behavior/code
Expected quirrel/blitz package to be installed under node_packages

Environment

• Quirrel version: 1.2.0
• Node/npm version: v14.16.0

At the moment, getting all jobs across all queues (as done by Quirrel UI), is quite expensive. It does a full Redis SCAN across the instance, which doesn't scale that well.
To counteract, we should store a set of all queues that a token has.

Bug Report

I am trying to run the example (adapted) from the documentation https://docs.quirrel.dev/recipes/processing-upload-in-background in local development but the code inside the Queue function doesn't run.

Current Behavior
The UploadedCSV record is created with success and call the csvProcessingQueue.enqueue is listed in https://ui.quirrel.dev/activity with the statuses scheduled/started/completed as expected, but the code inside (which I reduce to the minimum (only the await db.uplodadedCsv.delete({where: {id: uploadId}}) part) is not running.

Also, I don't know if it is also an expected behavior but no console.log are showing on the console either.

Input Code

Here is the most reduce sample it could provide :

in app/api/upload-image.ts

import imageQueue from "app/api/upload-queue"

const uploadImage = async (req, res) => {
    // The poiImageToUpload id 13 exist in the database
    await imageQueue.enqueue(13)

    res.status(201).send({ data: "Successfully created" })
}

export default uploadImage

in app/api/image-queue.ts

import { Queue } from "quirrel/blitz"
import db from "db"

export default Queue(
  "api/image-queue",
  async (id: number) => {    
    await db.poiImageToUpload.delete({ where: { id } })
  }
)

Environment

• Quirrel version: 1.3.4
• Node/npm version: 14.16.0/6.14.11
• Blitz version 0.33.0 (and also tried 0.32.1)

Bug Report

Current Behavior
When I set the input parameter of the Queue function to a date type and pass in a date to the enqueue function, the type of the parameter is a string.

Input Code

• REPL or Repo link if applicable:

export default Queue(
  "api/generateRosterQueue", // the path of this API route
  async (date: Date) => {
    console.log(`Generate roster task executing for date: ${typeof date}`)
  }
)

await GenerateRosterQueue.enqueue(new Date())

Expected behavior/code
I expect the tyoeof date to return a Date type, but it returns a string

Environment

• Quirrel version: 1.2.0
• Node/npm version: 15.8.0 / 7.5.1

Additional context/Screenshots

Bug Report

Current Behavior
Updating to version 1.4.3 I'm getting the error Type error: Cannot find name 'window' building my project. Could be related to this issue lquixada/cross-fetch#104 unfortunately I can't found a workaround to make it works (like adding a resolution to the latest cross-fetch version).

Disabling all imports from Quirrel on my project import quirrel import { Queue } from "quirrel/blitz" it's building fine.

Environment

• Quirrel version: v1.4.3
• Node/npm version: Node v15.13.0

See the following GIF. The server constantly tried to connect, and "attach" to server.

I am running on WSL2, but it should not make a difference. I can see the UI is connecting to localhost:9181 and I see the network requests are returning OK and resolving.