arch199 / crontrack Goto Github PK

Running the startmonitor command causes the CPU to spike due to this:

Should be easy to reproduce.

Since we're calling the startmonitor command from the command line (or cron), there's no need for any threading, since its own thread is already specifically just for this task. When running the monitor in tandem with the runserver command it made sense, but that wouldn't be recommended for production. It can be convenient while developing so that you don't need to run a separate command, but it should probably be disabled unless in DEBUG mode (or removed entirely for consistency with prod).

I've got a test job to run every 5 minutes w/ a 10 minute notification window, and when I purposely skip notifications, I correctly get notified after 15 minutes, but then it re-sends an alert each 15 minutes. Dead jobs should only alert once, unless they come back alive and go dead again.

Potentially we could have an escalation mechanism so that it would send one alert at first, but then we could configure a second or third notification if it goes X more time periods without acknowledgement. Escalation could start sending to other email addresses or text messages after reaching certain thresholds. Would have to flesh out details on how this could work.

Currently, emails are showing the times in UTC, but the web interface correctly shows times in my own time zone.

I set up a task to run every minute (* * * * *), with a time window of 5 mins. I am pinging it each minute, but I get an alert every 5 minutes. The next_run doesn't get updated until it fails, so I think that value needs to get recalculated each time a notification happens.

Could be useful to have a button or action somewhere to "reset" a job, clearing its notification/failed times. When editing a job, it would be good to have "Save" and "Save & Reset", so if you're changing the cron window, you can easily reset it.