GithubHelp home page GithubHelp logo

agenda / agenda-rest Goto Github PK

View Code? Open in Web Editor NEW
139.0 19.0 44.0 1.12 MB

Scheduling as a Service

License: MIT License

JavaScript 99.53% Dockerfile 0.47%
agenda scheduling agenda-rest cron rest

agenda-rest's Introduction

npm version Build Status Slack Status

agenda-rest

Scheduling as a Service, based on Agenda

Assuming all job types could be thought of as REST endpoints, scheduling could be offered as a service. agenda-rest does just that, introduce a URL, name it, agenda-rest will call it on the times that you specify.

Installation

Install agenda-rest as a global package

npm install -g agenda-rest

Usage

To launch the agenda-rest server, use the command line interface specifying the database host name and the database name

agenda-rest --dbhost localhost --dbname agenda

Command Line Interface options

Options Description
-d, --dbname [optional] Name of the Mongo database, default is agenda
-h, --dbhost [optional] Mongo instance's IP or domain name, default is localhost
-u, --dburi [optional] Full Mongo connection string. If specified, will override --dbhost, --dbname
-p, --port [optional] agenda-rest server port, default is 4040
-k, --key [optional] x-api-key to be expected in headers. If not specified, access to agenda-rest server would be unauthenticated
-t, --timeout [optional] Timeout for request duration, default is 5000 ms
-a, --agenda_settings [optional] A JSON string containing additional agenda settings. For example '{ "processEvery": "30 seconds" }'

APIs

API Documentation

API Documentation (Postman Generated) available at https://explore.postman.com/templates/4883/agenda-rest

GET /api/job

Get a list of defined jobs

  • Method: GET

POST /api/job

Defines a new category of jobs

  • Method: POST
  • Data:
{
    name,           // New job type's name
    url,            // koa-router style url
    method,         // (optional) Request type, default: POST
    callback: {     // (optional) to call with response after invocation
        url,
        method,
        headers
    }
}

PUT /api/job/:jobName

Updates definition of a job category

  • Method: PUT
  • Data: same as POST /api/job

DELETE /api/job/:jobName

Deletes job definition and cancels occurrences

  • Method: DELETE

POST /api/job/once & POST /api/job/every

Schedule a job for single or multiple occurrences

  • Method: POST
  • Data:
{
    name,           // Name of the type to create the instance from
    interval,       // Interval in which job should be invoked (human-interval, can also be a date string for 'once')
    data: {         // (optional) default: {}
        headers,    // Http headers, e.g. { Authorization: '<token>' }
        params,     // An object i.e. { param1: 'value1' } used to replace path parameters `http://mydommain.com:3333/test/:param1` => `http://mydommain.com:3333/test/value1` notations in the job definition's url.
        query,      // An object i.e. { foo: 'bar', baz: 'qux' } used to create query parameters (http://mydommain.com:3333/test/value1?foo=bar&baz=qux)
        body        // Accompanying data sent along the request
    },
    options: { // (optional) Enables passing options to the `every` method in agenda as documented [here](https://github.com/agenda/agenda#repeateveryinterval-options)
      timezone, // Specify the job execution timezone.
      skipImmediate // Don't execute job immidiatly default is `false`.
    }
}

Callback, if present, would be invoked by the following object:

{
    data: {
        // passed data object, same as above
    },
    response        // response from invocation
}

POST /api/job/now

Like once and every, though without interval. Executes the job now.

POST /api/job/cancel

Cancels (not to be confused with 'delete') any jobs matching the query

  • Method: POST
  • Data: Mongo query
{
  name: "foo"
}

agenda-rest's People

Contributors

abhijitmehta avatar arvant avatar chickencoder avatar dyoxyne avatar farhoud avatar geosp avatar keyvan-m-sadeghi avatar omgimalexis avatar qlonik avatar renovate-bot avatar renovate[bot] avatar sampathblam avatar simison avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

agenda-rest's Issues

Command line flag does not change DBhost

agenda-rest -h 172.18.0.3 -d Agenda

throws the following error:

MongoError: failed to connect to server [localhost:27017] on first connect [MongoError: connect ECONNREFUSED 127.0.0.1:27017]
    at Pool.<anonymous> (/usr/local/lib/node_modules/agenda-rest/node_modules/mongodb-core/lib/topologies/server.js:336:35)
    at Pool.emit (events.js:180:13)
    at Connection.<anonymous> (/usr/local/lib/node_modules/agenda-rest/node_modules/mongodb-core/lib/connection/pool.js:280:12)
    at Object.onceWrapper (events.js:272:13)
    at Connection.emit (events.js:180:13)
    at Socket.<anonymous> (/usr/local/lib/node_modules/agenda-rest/node_modules/mongodb-core/lib/connection/connection.js:189:49)
    at Object.onceWrapper (events.js:272:13)
    at Socket.emit (events.js:180:13)
    at emitErrorNT (internal/streams/destroy.js:64:8)
    at process._tickCallback (internal/process/next_tick.js:114:19)

Looks like it is still trying to connect to the localhost DB. Or am I doing something wrong?

Adding Jobs Returns Duplicate ObjectID

Hey there. We are experiencing some strange behavior and we are unsure if it is our own fault from lack of understanding or something to do with agenda-rest.

Error:

WriteError({"code":11000,"index":0,"errmsg":"E11000 duplicate key error collection: agenda.jobDefinitions index: id dup key: { : ObjectId('5afc675a3f77ac1f5715b6ba') }","op":{"name":"test111","url":"192.168.1.17:1880/relayon","method":"POST","_id":"5afc675a3f77ac1f5715b6ba"}})

What happens?

We start agenda-rest.
We submit a job.
We verify the job in jobDefinitions. It is there.
We submit another job.
We receive a duplicate ID warning.
We restart agenda-rest.
We submit a job, it works.
We submit again, returns duplicate ID error.

Whenever the first job is submitted, it becomes the only ObjectID that is attempted to be used while agenda-rest is running. Inserting directly to MongoDB works just fine.

Regards

Cannot find module './settings'

[email protected] start C:\Users\hp\Desktop\SocialNetwork\client

react-scripts start

internal/modules/cjs/loader.js:883
throw err;
^

Error: Cannot find module './settings'
Require stack:

  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules@nodelib\fs.walk\out\index.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\fast-glob\out\readers\stream.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\fast-glob\out\providers\async.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\fast-glob\out\index.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\globby\index.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\react-dev-utils\globby.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\react-scripts\scripts\utils\verifyTypeScriptSetup.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\react-scripts\scripts\start.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
    at Function.Module._load (internal/modules/cjs/loader.js:725:27)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object. (C:\Users\hp\Desktop\SocialNetwork\client\node_modules@nodelib\fs.walk\out\index.js:7:20)
    at Module._compile (internal/modules/cjs/loader.js:1063:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Module.require (internal/modules/cjs/loader.js:952:19) {
    code: 'MODULE_NOT_FOUND',
    requireStack: [
    'C:\Users\hp\Desktop\SocialNetwork\client\node_modules\@nodelib\fs.walk\out\index.js',
    'C:\Users\hp\Desktop\SocialNetwork\client\node_modules\fast-glob\out\readers\stream.js',
    'C:\Users\hp\Desktop\SocialNetwork\client\node_modules\fast-glob\out\providers\async.js',
    'C:\Users\hp\Desktop\SocialNetwork\client\node_modules\fast-glob\out\index.js',
    'C:\Users\hp\Desktop\SocialNetwork\client\node_modules\globby\index.js',
    'C:\Users\hp\Desktop\SocialNetwork\client\node_modules\react-dev-utils\globby.js',
    'C:\Users\hp\Desktop\SocialNetwork\client\node_modules\react-scripts\scripts\utils\verifyTypeScriptSetup.js',
    'C:\Users\hp\Desktop\SocialNetwork\client\node_modules\react-scripts\scripts\start.js'
    ]
    }
    npm ERR! code ELIFECYCLE
    npm ERR! errno 1
    npm ERR! [email protected] start: react-scripts start
    npm ERR! Exit status 1
    npm ERR!

C:\Users\hp\Desktop\SocialNetwork>npm start --prefix client

[email protected] start C:\Users\hp\Desktop\SocialNetwork\client
react-scripts start

internal/modules/cjs/loader.js:883
throw err;
^

Error: Cannot find module './settings'
Require stack:

  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules@nodelib\fs.walk\out\index.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\fast-glob\out\readers\stream.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\fast-glob\out\providers\async.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\fast-glob\out\index.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\globby\index.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\react-dev-utils\globby.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\react-scripts\scripts\utils\verifyTypeScriptSetup.js
  • C:\Users\hp\Desktop\SocialNetwork\client\node_modules\react-scripts\scripts\start.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)

node version:v14.15.3
npm version:6.14.9

More maintainers for agenda-rest + npm package

Would be cool if we can have a couple more maintainers with write access to the repo as well as the npm package for faster publish, as it seems I've been a bit neglectful in this regard.

Anyone feeling like it, your help is very much appreciated.

I'd need npm usernames.

@geosp
@sampathBlam

Params Does not Work when is pased to schedule a Job

Im Trying to create a Job with a Basic URL and schedule this Job using params to not repeat the same Job with different Params, but I check an the params are not sending like that into the schedule.

http://localhost:4040/api/job/now

This is the Body I sent:
{
"name": "SapUp",
"data": {
"params":
{
"vm_name": "NVD-PW-CA5-SAP-VT-11",
"instance": "nn",
"ticket":"TICKETTEST08",
"mail":"[email protected]"
}
}
}

This is the Job Definition Im Using

{
"_id" : ObjectId("5d1638e15f48de2af1aace10"),
"name" : "SapUp",
"url" : "https://webstation.cositas.com:3700/api/sap_up",
"method" : "POST"
}

the errror Im Recieving is

{
"_id" : ObjectId("5d163962bbc6042b04da0fe4"),
"name" : "SapUp",
"data" : {
"params" : {
"vm_name" : "NVD-PW-CA5-SAP-VT-11",
"instance" : "nn",
"ticket" : "TICKETTEST08",
"mail" : "[email protected]"
}
},
"type" : "normal",
"priority" : 0,
"nextRunAt" : null,
"lastModifiedBy" : null,
"lockedAt" : null,
"lastRunAt" : ISODate("2019-06-28T15:59:32.224Z"),
"failCount" : 1,
"failReason" : "406 - {"id":406,"msg":"Debe ingresar vm_name"}", (Is Not recieving this Param)
"failedAt" : ISODate("2019-06-28T15:59:32.379Z"),
"lastFinishedAt" : ISODate("2019-06-28T15:59:32.379Z")

Is this an Issue of the agenda-rest or Im using wrong the npm.

Best regards
Andres

How to repeat job at Friday at 5am for every week

Hi, using agendash, i can configure job like this:

agendash repeat job every week

how can i do the same using /api/job/every?
i am trying something like this:

{
    "name": "do-something",
    "interval": "At 05:00 on Friday",
    "data": {"name": "Your metadata goes here..."}
}

but it seem not schedule correctly.

How to introduce logging

Introduce logging.

For example, if I want to count the number of times my API is run, how should I go about going so with agenda-rest? How about a system that logs the numbers of iterations an API is called?

Thank you, :)

POST to /job/every consistently 'overwrites' last job instance with same name

Hi, I've setup agenda-rest after a year using agenda in a monorepo. For some reason, when I post to /job/every instead of creating a new job in the db, the most recent job is removed and a new job is created with a new mongo _id.

I first noticed it in my app, but I've replicated the behavior with postman and I'm now pretty sure I've isolated it to the agenda-rest app I have running on heroku.

What am I doing wrong? Missing parameter? Uniqueness issue? Seems like it could be an issue with the job definition but there's not much to mess up there.

Any help would be much appreciated!

The job fails when specifying port number, parameters and query strings.

I experienced several job failures when specifying a port number, param values and query parameters.
Here is an snipped of code that exemplifies the issue:

let scheduleEvent = async ({ sourceID, frequency, path }) => {
  let {
    scheduler: { uri, port, agendaURI },
  } = config
  if (_.size(frequency)) {
    try {
      // Delete event if it exists.
      await httpClient.delete(`${agendaURI}api/job/${sourceID}`)
    } catch {}
    let payload = {
      name: sourceID,
      // interval: `${_.join(',', frequency)} * * * *`,
      interval: 'three minutes',
      data: {
        body: { scriptPath: `${path}/main.py` },
        // headers: { 'X-Nothing-None': 'NoNoNoNoNo' },
        // params: { id: 24, name: 'donkey' },
        // query: { foo: 'bar', baz: 'qux' },
      },
    }
    await httpClient.post(`${agendaURI}api/job`, {
      body: {
        name: sourceID,
        // url: `${uri}:${port}/launch/:name/:id`,
        url: `${uri}:${port}/launch`,
      },
      json: true,
    })
    await httpClient.post(`${agendaURI}api/job/every`, {
      body: payload,
      json: true,
    })
  }
}

Here is the agenda job JSON that shows the error message:

{
    "_id" : ObjectId("5e4b43cbe7137fe09a0a6bff"),
    "name" : "be51bb3af103899306432b9f156a6c5a",
    "type" : "single",
    "data" : {
        "body" : {
            "scriptPath" : "test/data/bids/us/fl/myflorida/main.py"
        }
    },
    "lastModifiedBy" : null,
    "nextRunAt" : ISODate("2020-02-18T01:57:22.096Z"),
    "priority" : 0,
    "repeatInterval" : "three minutes",
    "repeatTimezone" : null,
    "lockedAt" : null,
    "lastRunAt" : ISODate("2020-02-18T01:54:22.096Z"),
    "failCount" : 1,
    "failReason" : "Cannot destructure property 'keys' of 'obj' as it is undefined.",
    "failedAt" : ISODate("2020-02-18T01:54:22.097Z"),
    "lastFinishedAt" : ISODate("2020-02-18T01:54:22.097Z")
}

define vs jobs

Hi!

I wanted to ask a question about the implementation just to understand the logic. The API /api/new creates a definition of a job and the callback and then /api/every or /api/schedule creates the jobs. I was wondering what the reason was not to just create the definition at the same time as the job is created? Why did you decide to create a table of definitions?

Thanks!

Tony

Master todo thread

I just want to list here quickly what could/should be done. Each topic could be its own issue/PR.

Janitorial

We'd love to have this repo in line with our other repos (see @agenda) and that mostly includes

  • using xo,
  • codebase in ES6,
  • tests with ava.
  • Add this to our Travis CI

I already wrote at Slack:

just a reminder that since that project has been independent, they might be opinionated about coding style, libraries to use etc.

Nicer to have everything in line (xo etc) under the org, but should not become a blocker if they strongly prefer not to.

;-) Would love to hear your thoughts on that @keyvan-m-sadeghi

Bigger picture tasks / questions

  • map out what requirements Agendash has from the API and if this library matches them.
  • map out if this library works with the latest Agenda (1.x).
  • map out Agenda methods
  • Write Agendash to use this package instead its built-in API
  • if we need to make breaking changes here, can we or do we need to maintain backwards compatibility (might not be much work)?
  • Should we add versioning to the API (/v1/ in routes etc)?

What do you think?

SyntaxError: Unexpected token *

When I do npm run start, I am getting this error. Any ideas?

/***************************************************************************************************/
async * Symbol.asyncIterator {
^

SyntaxError: Unexpected token *
at createScript (vm.js:80:10)
at Object.runInThisContext (vm.js:139:10)
at Module._compile (module.js:599:28)
at Object.Module._extensions..js (module.js:646:10)
at Module.load (module.js:554:32)
at tryModuleLoad (module.js:497:12)
at Function.Module._load (module.js:489:3)
at Module.require (module.js:579:17)
at require (internal/module.js:11:18)
at Object.pythonic (C:\Users\joe.h\Desktop\agenda_rest\agenda-rest\dist\index.js:564:18)

/***************************************************************************************************/

Provide information about the CLI arguments in the documentation

Do you guys think if we need information about the command line arguments that we give to agenda-rest, in the documentation? At this point, we can get to know about them only by having a look at cli.js. Let me know if this makes sense. I can work on a PR for that.

Not maintained anymore?

I was looking for something like this for a project that I am working on. Is this not actively maintained anymore? I only see renovate-bot commits for around 2 years now.
I would love to contribute and help maintain this project if needed, and if there are issues that need to be worked on.

Migrate tests to Mocha

  • Agenda is now using mocha for testing instead of ava.
  • The latest version of ava is not working with our current tests.

Missing documentation on how to define jobs

For the life of mine I can not figure out how to use the POST /job endpoint to define a job or load my already defined jobs into the agenda-rest instance.

I've spent about four hours reading the documentation and reading through the source code and I completely do not understand how to use this endpoint.

In the docs it says the endpoint takes a url, but I have no idea what should be under that URL and in what format. How do I provide the JS source code of my job? Or am I getting this completely wrong?

Also, I'd love to preload my jobs from source instead of having to define them with endpoints. Is this possible with this library?

This essentially stops me from using the library because I do not know how to define jobs and the documentation is as helpful as saying: url. I don't even know what that means.

After reading the source code I can see that the only thing that is being done in defineJob is just making a HTTP request to the url and then calling a callback. I don't understand where the my source code of my job supposed to be taken from.

Please help and I suggest to add much more instructions on how to define a job which is de facto the first endpoint you want to deal with when trying to use this library.

This is a shame as I would love to use it - the rest seems great. But until I get some answers I will have to rely on a custom written simple express www server with pure agenda.

I'm happy to produce documentation content if maintainers would explain how to use this endpoint.

Must run on port 8080? Failure ECONNREFUSED 127.0.0.1:8080

When i run agenda-rest -u <mongo URI> -p 8082 i get the following failure on jobs

"failCount" : 2, 
"failReason" : "failure in callback: Error: connect ECONNREFUSED 127.0.0.1:8080"

my job definitions do not use port 8080 so i was very confused.

i then ran agenda-rest on port 8080 to see if it was failing to connect to the agenda-rest API (expecting it on 8080) and the job worked!

i dont see anywhere in the code this hardcoded expectation of agenda-rest running on 8080. where is that coming from and how can it be modified to match the port passed in -p CLI option?

also a related error is the callback.url seems to invoke the agenda-rest API at that path. the job succeeded but callback failed (i had set callback.url to be another endpoint of mine to confirm how it worked). it looks like it ignores the host/port and just tries to do <agenda-rest>/path. is that expected?

Agenda-rest has been slow lately

I have been trying to schedule jobs from agenda-rest but it seems like it has been slow lately. It takes painful amount of time to schedule a job - more than ~200seconds. What can be the issue? Am I missing something here?

Cannot find module '../settings'

Following instructions in README, did this:

npm install -g agenda-rest
agenda-rest --dbhost my.ip.here --dbname agenda

and get the following:

module.js:538
throw err;
^

Error: Cannot find module '../settings'
at Function.Module._resolveFilename (module.js:536:15)
at Function.Module._load (module.js:466:25)
at Module.require (module.js:579:17)
at require (internal/module.js:11:18)
at Object. (/usr/lib/node_modules/agenda-rest/dist/index.js:1:1547)
at o (/usr/lib/node_modules/agenda-rest/dist/index.js:1:422)
at Object. (/usr/lib/node_modules/agenda-rest/dist/index.js:1:4530)
at o (/usr/lib/node_modules/agenda-rest/dist/index.js:1:422)
at Object.defineProperty.value (/usr/lib/node_modules/agenda-rest/dist/index.js:1:843)
at /usr/lib/node_modules/agenda-rest/dist/index.js:1:852

Am I missing something that's not included in the 'npm install' ?

OS: Ubuntu 16.04 LTS
Node: 8.9.3
NPM: 5.8.0

Wanna join the gang?

Hello @keyvan-m-sadeghi ๐Ÿ‘‹

You've done some impressive work here! ๐Ÿ‘ ๐Ÿ‘

We're working on Agenda and Agendash โ€” latter one has built-in API like yours. It would be awesome if the API would be separate (like your package), somewhat "official" and Agendash could use that then instead.

What do you think? Would you be interested helping out with such project? :-) We could move this package to @agenda org and it would get more attention from the awesome Agenda community.

We have Slack channel if you wanna join and discuss this more. :-)

Have a good one!

How to secure the rest endpoint

Hello,
How can I secure agenda rest endpoint so that only authorized people can schedule requests?
There should be an apikey but I don't find anything in doc.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.