pschlan / cron-job.org

cron-job.org Open Source project

cron-job.org

Structure

• database contains the MySQL database structure.
• chronos is cron-job.org's cron job execution daemon and is responsible for fetching the jobs.
• protocol contains the interface definitions for interaction between system nodes.
• frontend contains the web interface
• statuspage contains the status page UI
• api contains the server API used by web interface and status page UI.

chronos

Concept

chronos checks the MySQL database every minute to collect all jobs to execute. For every minute, a thread is spawned which processes all the jobs. Actual HTTP fetching is done using the excellent curl multi library with libev as library used to provide the event loop. Together with the c-ares resolver this allows for thousands of parallel HTTP requests.

cron-job.org supports storing the job results for the user's convenience. This can quickly lead to I/O bottlenecks when storing the result data in a MySQL database. (Which also has the downside that cleaning up old entries is extremely expensive.) To solve this issue, chronos stores the results in per-user per-day SQLite databases. Cleaning up old entries is as easy as deleting the corresponding day's databases.

The whole software is optimized on performance rather than on data integrity, i.e. when your server crashes or you have a power outage / hardware defect, the job history is most likely lost. Since this is volatile data anyway, it's not considered a big issue.

chronos can now run on multiple nodes. Each node requires an own MySQL server/database and stores its own jobs. The host running the web interface also manages the user database and an association between job and node. The web interface can create, delete, update and fetch jobs and job logs from the particular node via a Thrift-based protocol defined in the protocol folder.

Prerequisites

In order to build chronos, you need development files of:

• curl (preferably with c-ares as resolver and libidn2 for IDN support)
• libev
• mysqlclient
• sqlite3
• thrift (compiler and libthrift)

To build, you need a C++14 compiler and cmake.

Building

1. Create and enter a build folder: mkdir build && cd build
2. Run cmake: cmake -DCMAKE_BUILD_TYPE=Release ..
3. Build the project: make

Running

1. Ensure you've imported the DB scheme from the database folder
2. Customize chronos.cfg according to your system (especially add your MySQL login)
3. Execute ./chronos /path/to/chronos.cfg

API

The API is written in PHP and needs to be hosted on a webserver (cron-job.org uses nginx with php-fpm). It is used by the console and the status page UI.

Prerequisites

• nginx with php-fpm (PHP 7)
• Optionally, a redis instance to support API call rate limiting

Getting started

• Copy the api/ folder to your webserver
• Create a copy of config/config.inc.default.php as lib/config.inc.php and customize it according to your environment

Notes

• When changing the thrift protocol, don't forget to re-compile the PHP glue code and copy it to lib/protocol/. When committing, include the updated PHP code. Currently, this is a manual step.

Frontend

The frontend is written in JavaScript using React and material-ui. You need npm to build it.

Prerequisites

• Node.js

Getting started

• Go to the frontend/ folder
• Install all required dependencies by running npm install
• Create a copy of src/utils/Config.default.js as src/utils/Config.js and customize it according to your environment
• Run the web interface via npm start

Status Page Frontend

The status page frontend is written in JavaScript using React and material-ui. You need npm to build it.

Prerequisites

• Node.js

Getting started

• Go to the statuspage/ folder
• Install all required dependencies by running npm install
• Create a copy of src/utils/Config.default.js as src/utils/Config.js and customize it according to your environment
• Run the web interface via npm start

Example Environment (using Docker Compose)

To quickly start an example environment of most of the cron-job.org system, you can use docker-compose:

• Initialize/update submodules: git submodule init && git submodule update
• Copy .env.example to .env and open it in a text editor
• Change the variables in .env as desired. As an absolute minimum, fill the *_SECRET variables with randomly generated secrets and specify a SMTP server in SMTP_SERVER. No authentication is used for the SMTP session, so ensure that the SMTP server is allowing relaying for your machine's IP address.
• Start via docker compose up and wait until all containers are built and all services have been started. This can take a while, especially on first run.
• Open http://localhost:8010/ in your browser (assuming you kept the default port and host name settings).

Important: The Docker environment contained in this repo is intended as an example / development environment and is not tailored for production usage, especially with regard to security.

The following containers will be started:

• mysql-master for the master service database which also stores users, groups, job -> node associations etc.
• mysql-node for the node service database. Used to store per-executor information like job details with their schedules, etc.
• redis used as a cache for certain features like rate limiting.
• api for the PHP-based API backend used by the frontend. Uses php-fpm.
• frontend which hosts the built frontend code.
• wwww as the frontend nginx-powered HTTP server which connects to frontend (HTTP reverse proxy) and api (FastCGI).
• chronos which runs chronos in a combined master and node service mode.

General notes

• We strongly recommend to build curl using the c-ares resolver. Otherwise every request might spawn its own thread for DNS resolving and your machine will run out of resources very soon.
• Before running chronos, ensure that the limit of open files/sockets is not set too low. You might want to run ulimit -n 65536 or similar first.
• If data integrity is not important for you, we highly recommend to set innodb_flush_log_at_trx_commit=0 and innodb_flush_method=O_DIRECT in your MySQL config for best performance. Otherwise the update thread (which is responsible for storing the job resuls) might lag behind the actual job executions quite soon.
• Parts of the source are quite old and from early stages of the project and might require a refactoring sooner or later.

cron-job.org's Issues

IDN url encoding issue

There seems to be an encoding issue with international domain names.
URL strings with non-ascii characters that are passed to the chronos HTTPRequest class cannot be invoked and result in an execution error. However, the Punycode representation of the same url name seems to work fine.

API for creating cron jobs?

Is there any API for creating cron jobs? I want to set cron jobs from my server itself as the job execution date will not be fixed.

Timezone / Documentation

I can modify hours of execution, but I don't get in which timezone I'm scheduling.
Maybe worth adding it to the doc, or (even better) let user choose his timezone

Add UI for annual recurring tasks

Currently this has to be done via the custom option. It would be good to standard UI element for it.

Add test run in web interface

Allow to test-fetch the job and show the results with detailed diagnosis information.

Allow pasting/parsing of crontab lines

Allow pasting/parsing crontab lines to add a cronjob. This should automatically detect popular wget and curl commands to retrieve URLs.

Appears to be falsely seeing a 500 error

A cron job is getting a text response and it's being seen as a 500 error - you can see the errors in https://cron-job.org/en/members/jobs/details/?jobid=719606 - I guess you can access this?

Add newsletter option

To keep users up to date (e.g. regarding IP address changes), it might make sense to add an (opt-in) newsletter.

No request body for http DELETE method

Would it be possible to allow a request body to be used for other HTTP verbs like DELETE? Many REST APIs uses DELETE with a json body to specify parameters like {"delete_all":true}.

website down?

Failed to connect to MySQL database backend

HTML files

Hi,
I am looking for the frontend files of the project but couldn't find it. Is it possible someone to show the file, it would be great if you can help me. Thanks

New feature: Retries on failure

I'm calling a lambda from this service which typically has to wakeup from cold freeze. Normally this works just fine within the cron-job.org timeout, but sometimes it doesn't. Just doing a request a few seconds later would work instantly.

There fore I'd like the option like this:

• enable retries
  • num of retries:
    • 1
    • 2
    • 3
  • delay retry (max 60s): 10 s

Just adding a cron job a few seconds later doesn't cut it, btw, because I want the job to be executed just once per day.

Support trigger jobs which do not wait / fail on timeout

website seems not execute the jobs anymore

the last success execute time is Yesterday, 22:03:59 (GMT+8)

A feature suggestion, spread out the schedule.

Not really an issue, but I have many users who I recommend to use your site to automate interaction with my service. Most of them choose to run the job "every 5 minutes" because it is convenient option.

The side effect of this is that I get spikes every 5 minutes and I noticed that sometimes this results in an error "Last status: Failed (503 Service Temporarily Unavailable)", I guess I need to upgrade my server :-)

In any case, I do wish to suggest something which may prove useful to improve quality of service: When a user chooses "every X minutes", cron-job service could generate a (random number % X) and use it to spread out user requests.

So for example, lets say we have 100 users, all of who chose to run the cron-job every 5 minutes. Instead of having all of them run at 1PM, 1:05PM, 1:10PM... each one would get a random number (at the time a cron-job is created) from 0 to 4 which would be used to offset their timings so that 20% of them run at original 1:00, 1:05, etc; another 20% would run at 1:01, 1:06, etc; another 20% would be at 1:02, 1:07... etc ...

A good idea would be to give users an option to override this, so they have an option of locking the offset at 0 or 3 or ... , but that option is not enabled by default.

Edit: As an alternative, it would be nice to give a user an option to specify the timings as a string. This could be a standard cron (link) formatted string or something of your own design. This way I could analyze my server usage and provide a suggested configuration when referring users to use your service. This would be essentially another way to specify a "User Defined" timings.

Thank you very much! If there is anything I can do to help with coding and / or testing this, please let me know.
Gene

Treat 20x status codes as success

20x status codes should be treated as success.

Response data too big on cron-job.org

I have some important elements to see in the response... but every execution it returns me Response data too big. Can you increase data size of response ? Also because the status is 200 but appears /!\ in executed column so the cronjob deactivated itself after many execution and I have to active it manually again...
Can you update this and also add a setting for increase timeout?

Add time zone selection with auto detection to signup form

Doesn't seem to send emails - the cronjob will be disabled because of too many failures

I get emails for
execution of the cronjob fails and
execution of the cronjob succeeds after it failed before
but not for
the cronjob will be disabled because of too many failures
and wondered if it's just me?

bot

..❤

Add

http://sajanmadh5.temp.swtest.ru

Make timezone configurable per job (take default value from user settings)

Resolve redirections when adding jobs

https://cron-job.org/ Weird cannot edit any cron ?

I try to edit any cronjobs on the cronlist but I always come out with only one cronjob or not the specific cron that I want to edit.

Please evaluate. Thank

Feature suggestion schedule cron weekday (Monday to Friday)

Hi, it would be amazing if you allow to set the cron every weekday

Send notification emails in user language

Currently all notifications are sent in German. Instead, they should be sent in the language of the last user login.

Feature suggestion import/export

Hi, I've created several jobs, would there be a way to export my cron jobs?

I'm going to install it on my own server so it would be nice to import all my cron jobs.

Is there going to be an import/export function in the future?

thanks

Support multiple executor nodes

Instagram

Dns error

Geting this error.. any ideas? My website is active and visable.

Could not resolve host: Could not resolve host (Could not contact DNS servers)

Support PATCH requests

I would like to see support for PATCH requests please. This will allow me to use cron-job.org to control my Heroku app using the API. Thanks!

Response Data Too Big on Cron-Job.org

Our cron jobs are failing with "response data too big" errors. Is there something we can do to fix? We are using cron jobs to import Airbnb calendars to booking calendar plugin on this site royalbluechalets.com.

I tried sending a message to [email protected] but the message did not go through. Got this message "The response from the remote server was:
550 No such local user, unauthenticated relaying denied"

Thank you

Pre-fetch jobs for next minute to reduce latency

Is there any plan fot the Web Interface?

Hi,

the deamon is now running on my Server. But i don't now insert Crons in my MySQL Table (whitch format). Exist any Tutorial?

Custom Repeat Interval

How would I create a job that executes once every 1 hour and 50 minutes?

add access to others for my cron jobs

Hi,

is it possible to add access to my cron jobs to other users ?

thanks

Add feature to re-send activation links

In case an activation link is not received, the user should be able to trigger a re-submission of the email, limited to a certain number of times (e.g. 3 times).

copy/clone feature?

Is it any plans to add "clone cron-job" feature?

Cronjob execution: Response data too big

hi,
I am using a html file .
my file execution time is 35 second.
my file is not complete executed and give Response data too big error.
please help me to solve this issue

(solved) Include failure details in notification email

Love your service and recommend it to everyone who needs a web based cron-job, but your failure notifications are unnecessarily vague.
It would be great if response headers and body are included within cron-job failure notification.

The problem is that only the last 20 responses are available for examination and by the time I see a failure notification, it is too late to go back and check what went wrong.

Alternately, it would be nice to dedicate a certain number of responses to failures. For example instead of saving the last 20 responses, it would be possible to save the last 15 responses + 5 failures. A previously recorded failure should not be overridden by a successful response, but can be overridden by newer failure or your standard 2 day expiration. Ideally if no failures are present, normal responses are recorded instead.

I would be willing to help code the changes if they can be implemented within cron-job.org which is currently being used by many of my referrals.

Thank you,
Gene

Implement cron job validation

To minimize abuse, cron job hosts should be validated by either sending a confirmation email to the domain owner or uploading a validation file.

Allow grouping jobs in projects / folders

Add full cronjob edit UI when creating cronjobs

Currently the cronjob has to be created then edited to set request type to POST and to add headers and post body.

Add single shot jobs

wget/curl?

Ignore this

Import/Export feature

Will there be a feature to import/export the cron-jobs?
Thanks

Server monitoring

Add a feature to monitor servers and send emails or SMS messages (push?) when failing. Allow checking by HTTP requests, checking whether a port is open and/or ICMP / ping.

Run job button (Interface Web)

Could you guys add a "Run Job" button on the web interface, to force it to run asap. :)

Support time zones

Respect the user's time zone in the web interface and when executing jobs. Add a time zone setting in the user account settings.

Fix zero time stamp in notification emails (01/01/1970)