mufaka / kronomata

KronoMata is a cross platform scheduled job runner.

KronoMata

KronoMata is a cross platform scheduled job runner. Scheduled jobs run implementations of IPlugin on configured hosts at configured recurrence intervals.

Docker

Docker images are available for the Web application and Agent. They can be run together with the following Docker compose:

version: '3.4'

services:
  kronomata.web:
    image: billnickel/kronomata:web
    environment:
      - ASPNETCORE_ENVIRONMENT=Release
    ports:
      - "5015:5002"	  

  kronomata.agent:
    image: billnickel/kronomata:agent 
    environment:
      - KronoMata__APIRoot=http://kronomata.web:5002/api/

Quick Start

KronoMata has 3 main components. The web application for managing scheduled jobs, the agent which is responsible for running the scheduled jobs, and plugins that are the actual executables for scheduled jobs. KronoMata can run on Windows, Linux, or Mac OS as long as .NET 6 is installed.

Web Application

1. Extract the contents of the KronoMata.Web-.zip.
2. Optionally edit the configuration in appsettings.json. The defaults should be suitable for running immediately.
3. In a console, navigate to the web installation directory and run using 'dotnet KronoMata.Web.dll'
4. Navigate to the URL defined in the appsettings.json 'Urls' configuration value. By default you should be able to access http://localhost:5002/

** You can run the web app as systemd service on Linux, there are some notes in the wiki. On Windows you can use the built in Task Scheduler to create a task that runs on startup.

Agent

1. Extract the contents of the KronoMata.Agent-.zip.
2. Edit the KronoMata:APIRoot value in appsettings.json to point to the web server if you are not running the agent on the same machine as the web server.
3. Be sure that the PackageRoot folder exists. This is where the agent will attempt to store plugins. By default, there needs to be a PackageRoot subdirectory beneath the agent executable.
4. In a console, navigate to the agent installation directory and run using 'dotnet KronoMata.Agent.dll'

The agent will poll the web server every minute for scheduled jobs to run. The first poll will be 1 minute after the agent starts. If this is the first time an agent has polled the web application for scheduled jobs, it will be added as a host that is disabled. Navigate to the Hosts section in the web application and you should see your host added.

4. Click on the 'More info' link for the host, check the 'Enabled?' checkbox and click the Save button to enable the host.

** You can also run the agent as systemd service on Linux, there are some notes in the wiki for the web app that can be adapted for the agent as well. On Windows you can use the built in Task Scheduler to create a task that runs on startup.

Packages

Plugins are implementations of the KronoMata.Public.IPlugin interface. To schedule these plugins you need to upload them to the web server in a zip file. Note that a zip file can contain more than one implementation of IPlugin. The upload process will discover all instances and create the corresponding plugins so that they can be scheduled to run. The release contains a sample plugin that will just echo configured variables to the log.

1. Navigate to the Packages view in the web application.
2. Provide a Package Name on the 'Upload Package' form.
3. Drag and drop, or click to browse for, the KronoMata.SamplePlugin zip file and then click the upload button.
4. If the upload is successful, you will be redirected to the Plugins view. Click on the Echo Plugin to view the parameters that were defined.

Scheduling

Scheduling jobs can be done for enabled hosts only (or All Hosts but only enabled hosts will execute jobs).

1. Navigate to the 'Jobs' view and click on the 'Add' button.
2. Fill in the form as shown in the following screen capture.

** Choose the appropriate host and plugin.

3. Click on the 'Save' button.
4. Because the IPlugin implementation provided configuration parameters, you should be redirected to the Scheduled Job Configuration view.
5. Fill in both parameters as the IPlugin implementation defined them as required.

6. Click on the 'Save' button.

If all goes well, you should start to see logs appearing for the execution of the plugins on the agent at the configured schedule. Navigate to the 'History' view to see a complete log.

Clicking on a history row will provide a popup that shows more information about that execution.

** Note that the first execution in the above screen capture took considerably longer to run. This is because the plugin was downloaded and extracted as part of the first run. Subsequent executions use the agents local version to execute.

kronomata's Issues

ScheduledJob Host selection is too rigid

Right now you can either schedule a job to run on a single host or all of them.

1. Change the definition of ScheduledJob.HostId to a string variant data type. (rename to Hosts)
2. Allow for selecting multiple hosts when saving a ScheduledJob; save as comma separated list.
3. Update ScheduledJob listing by machine name to respect the list of hosts on ScheduledJob.

Show Host in Grid on Scheduled Jobs View

It would be nice to know which hosts are running the Scheduled Jobs. For Hosts you can tell what Scheduled Jobs they are running but not vice versa.

Could not load file or assembly System.IO.Compression, Version=6.0.0.0

After a period of prolonged running (unsure how long) unzipping package files fails for the Agent and the Web App. Restarting solves the issue for a period of time.

This appears to be specific to Ubuntu 22.04 with the following dotnet versions: (dotnet --info)
Microsoft.AspNetCore.App 6.0.22
Microsoft.NETCore.App 6.0.22

Ubuntu 20.04 with the 6.0.21 dotnet versions doesn't exhibit this behavior nor does Windows.

Implement an IPlugin for Agent Maintenance

Agents download packages in order to run plugins. If a package is deleted from the web application, it should also be deleted from the agents local store. Before implementing, brainstorm other use cases where this plugin can be used.

1. Call a new API method for getting a list of currently defined packages.
2. Delete local files that are not included in that list.

** Consider pros/cons of using the ScheduledJob list to do this. If packages are local and there are no current scheduled jobs for them, delete?

Implement an IPlugin for Job History Expiration

#10 Is a prerequisite.

1. Add an API endpoint for expiring Job History. Use global settings for the expiration parameters.
2. Create an IPlugin that calls the API endpoint.

Long running jobs (> 1 minute) will cause for missed execution of subsequent jobs.

The Agent ticks every minute to check for jobs to run. If a job runs for more than a minute (eg: backup job) then ticks will be missed until that job finishes.

Run Now ability for Scheduled Job

This needs to be thought out a bit more but the gist is as follows:

1. Create another table named RunNowJob that mimics ScheduledJob but also includes a flag for completed.
2. Create another table name RunNowHistory that mimics JobHistory but links to RunNowJob
3. RunNowJob should allow for selecting hosts in the same way ScheduledJob does.
4. A separate RunNowJob row should be created for each selected host.
5. UNION RunNowJob and ScheduledJob for API call that Agent uses to get a list of jobs.
6. UNION RunNowHistory and JobHistory results
7. Respect new tables when expiring history

** It think that RunNowJobs should have a default EndTime to prevent them from running repeatedly as well as defaulting to Frequency Minute, Interval 1.

Brainstorm for an easier way to implement this. I think adding a flag to ScheduledJob for RunNow might work as long as it ends up creating a ScheduledJob for each selected host. UI can enforce defaults when RunNow is selected as well as separating the display of ScheduledJobs into groups. If we do go this route, it's probably better to add a ScheduleType (Recurrence, RunNow, RunOnce).

Implement a Cache Data Store

This should be an IDataStoreProvider implementation.

1. The implementation should have a "backing data store" that persists data.
2. Caching should only be done for relatively static data (not JobHistory)
3. Implementation should work as a write-through cache.

Update the Web application to use this store.

Agent should attempt to create PackageRoot if it doesn't exist

If the PackageRoot directory doesn't exist, the Agent cannot download and extract plugins. By default, the PackageRoot is configured as a subdirectory of the agent so it should be creatable.

If a release does not contain the empty PackageRoot sub directory, an error similar to the following will occur.

One or more errors occurred. (Could not find a part of the path 'path to \PackageRoot\92bee86d-58a1-48e2-b9b8-99d3d3adf179.zip'.)

Daily job with specified hours and minutes only runs on the start date.

A job that was scheduled to run daily with specified didn't run at the correct time on days after the specified start date and time.

It was expected that the job would have run at 3:05, 3:10, 3:15, 3:20 and at those same minutes for the 6 o'clock hour as well.

Allow for Auto-Refresh on History

Only do this on the History view. This should be configurable on the grid and default to off, refreshing every 30 seconds.

Implement a set timeout function for refreshing the jsGrid at a 30 second interval. Add an icon in the card header for enabling / disabling this.

Delete for Package, PluginMetaData, Host, and ScheduledJob

Allow for deletion of Package, PluginMetaData, Host, and ScheduledJob. These are potentially cascading deletes but required in order to effectively manage the application.

Refactor Data Tests to Prevent Copy Paste Coding

The test cases for SQLite, in Test.KronoMata.Data.SQLite, were cut and paste from the tests written in Test.KronoMata.Data.Mock. Additional test cases were added for SQLite but not for the Mock implementation.

The IDataStoreProvider and supporting IDataStoreXxx interfaces define the expected behavior of implementations and those are what should be tested against.

Refactor the tests in a manner that allows for testing implementations of IDataStoreProvider with minimal code duplication. Use the SQLite tests as the template and then refactor Test.KronoMata.Data.SQLite and Test.KronoMata.Data.Mock to use the common tests.

Take the following test case that looks identical in both sets of tests:

public void Can_Create()
{
	var now = DateTime.Now;

	var scheduledJob = new ScheduledJob()
	{
		PluginMetaDataId = 1,
		HostId = 1,
		Name = "Name",
		Description = "Description",
		Frequency = ScheduleFrequency.Week,
		Interval = 2,
		StartTime = now,
		EndTime = now,
		IsEnabled = true,
		InsertDate = now,
		UpdateDate = now
	};

	_provider.ScheduledJobDataStore.Create(scheduledJob);

	Assert.That(scheduledJob.Id, Is.EqualTo(1));
}

There should be an elegant way to just pass the _provider to a shared template like the following:

public void Can_Create()
{
	var scheduledJob = SomeCommonScheduledJobCan_CreateMethod(_provider);
	Assert.That(scheduledJob.Id, Is.EqualTo(1));
}

Packages View Card Width Inconsistent

Update the Packages view to use col-12 for card widths to match the style of the rest of the views.

Manual Job History Expiration

1. Provide a means to purge Job History based on age and/or log size. IJobHistoryDataStore should provide a method for deleting Job History that accepts parameters for daysToKeep and maxHistoryRecords.

2. Global Configuration should have these two parameters defined with some reasonable defaults (14 days, 2500 items)

3. Add a section to the Settings view for manually running expiration.

End of Central Directory record could not be found.

Job History shows an 'End of Central Directory record could not be found.' error message when the package zip file cannot be extracted.

The zip file is corrupted when serving from a linux KronoMata.Web instance to a Windows KronoMata.Agent instance. The same zip file is not corrupt when the agent is running on linux.

All Hosts configuration for Scheduled Job doesn't work

A Scheduled Job with 'All' selected for Host doesn't appear to work. No jobs are run and when viewing the Host, the Scheduled Job doesn't appear.

Occasional SQLite Database Locked Failures

When an agent is running several jobs at the same time, it can cause a database locking issue because of the simultaneous requests to write the JobHistory (via the API call).

07:53:13 fail: KronoMata.Web.Controllers.AgentController[0] 
=> SpanId:64d28d8bfe22583a, TraceId:2af3454693e0c2e76487aeb5c1482021, ParentId:0000000000000000 => ConnectionId:0HMT8JPAB6OQS 
=> RequestPath:/api/Agent/history RequestId:0HMT8JPAB6OQS:00000007 
=> KronoMata.Web.Controllers.AgentController.Post (KronoMata.Web) Error creating JobHistory code = Busy (5), 
message = System.Data.SQLite.SQLiteException (0x87AF00AA): database is locked database is locked    
at System.Data.SQLite.SQLite3.Prepare(SQLiteConnection cnn, SQLiteCommand command, String strSql, SQLiteStatement previous, UInt32 timeoutMS, String& strRemain)    
at System.Data.SQLite.SQLiteCommand.BuildNextCommand()    
at System.Data.SQLite.SQLiteDataReader.NextResult()    
at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)    
at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)    
at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)    
at System.Data.SQLite.SQLiteConnection.Open()    
at KronoMata.Data.DbConnectionDataStoreBase.Execute(Action`1 action) in D:\Development\KronoMata\KronoMata.Data\DbConnectionDataStoreBase.cs:line 41    
at KronoMata.Data.SQLite.SQLiteJobHistoryDataStore.Create(JobHistory jobHistory) in D:\Development\KronoMata\KronoMata.Data.SQLite\SQLiteJobHistoryDataStore.cs:line 46    
at KronoMata.Data.InMemory.InMemoryJobHistoryDataStore.Create(JobHistory jobHistory) in D:\Development\KronoMata\KronoMata.Data.InMemory\InMemoryJobHistoryDataStore.cs:line 19    
at KronoMata.Web.Controllers.AgentController.Post(JobHistory history) in D:\Development\KronoMata\KronoMata.Web\Controllers\AgentController.cs:line 93

This does cause a JobHistory save to be missed the following screen capture shows only 6 history rows create for 7:53. It is expected that there are 7.

Version the API

The API root should contain a version (eg: /api/v1/) in order to support the ability to enhance the API without introducing breaking changes for existing agents.

API access protection.

All API endpoints need to be protected with a key and any controller method that returns data should require authorization.

GlobalConfiguration IsMasked and ConfigurationValue Password types should be encrypted at rest.

GlobalConfiguration and ConfigurationValues are stored in plain text. IsMasked and DataType.Password are only considered on the front end for hiding the values but they should be encrypted in the database.

Linux systemd service for Agent doesn't run

The agent service exits immediately when running as a systemd service on linux. The agent runs fine if started manually.

Service definition in /etc/systemd/system/KronoMata.Agent.service

[Unit]
Description=KronoMata Agent Application

[Service]
WorkingDirectory=/home/bnickel/KronoMata/Publish/Agent
ExecStart=/usr/bin/dotnet /home/bnickel/KronoMata/Publish/Agent/KronoMata.Agent.dll
SyslogIdentifier=KronoMataAgent

[Install]
WantedBy=multi-user.target

Starting service and checking status shows an inactive (dead) service.

bnickel@flash3r:/etc/systemd/system$ sudo systemctl start KronoMata.Agent
bnickel@flash3r:/etc/systemd/system$ sudo systemctl status KronoMata.Agent
● KronoMata.Agent.service - KronoMata Agent Application
	 Loaded: loaded (/etc/systemd/system/KronoMata.Agent.service; disabled; vendor preset: enabled)
	 Active: inactive (dead)

The full log shows it starting but then stopping.

sudo journalctl -u KronoMata.Agent.service	 
Aug 25 09:50:08 flash3r systemd[1]: Started KronoMata Agent Application.
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 dbug: Microsoft.Extensions.Hosting.Internal.Host[1] Hosting starting
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: KronoMata.Agent.PluginRunner[0] KronoMata Agent starting.
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 dbug: Microsoft.Extensions.Hosting.Internal.Host[3] Hosting stopping
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: Microsoft.Hosting.Lifetime[0] Application is shutting down...
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down.
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: Microsoft.Hosting.Lifetime[0] Hosting environment: Production
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: Microsoft.Hosting.Lifetime[0] Content root path: /home/bnickel/KronoMata/Publish/Agent/
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 dbug: Microsoft.Extensions.Hosting.Internal.Host[2] Hosting started
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 dbug: Microsoft.Extensions.Hosting.Internal.Host[3] Hosting stopping
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: KronoMata.Agent.PluginRunner[0] KronoMata Agent stopped.
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 dbug: Microsoft.Extensions.Hosting.Internal.Host[4] Hosting stopped
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: KronoMata.Agent.PluginRunner[0] KronoMata Agent stopped.
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 dbug: Microsoft.Extensions.Hosting.Internal.Host[4] Hosting stopped
Aug 25 09:50:09 flash3r systemd[1]: KronoMata.Agent.service: Succeeded.	 

Store dates as UTC

Job History RunTime and CompletionTime needs to be stored as UTC and then converted for display in the web app.

ScheduledJob dates will need to be thought out more before implementing because there might be a case where agents span multiple timezones and you want them to run at the same UTC time. But there are other use cases where you want the job to run at specific local times regardless of timezone (eg: fetch files from SFTP at 3:00 am).

Agent Verification of IPlugin Assembly

The KronoMata.Agent application should have a means to verify that the code being run is what the KronoMata.Web application expects. Right now it would be fairly easy to decompile a plugin, alter it in a malicious way, and then replace the existing one.

One option is to store a checksum on PluginMetaData and have the Agent check that before executing the plugin.

Linux Web service aborted

This happened when updating a ScheduledJob to run on 'All' hosts.

sudo systemctl status KronoMata.Web

● KronoMata.Web.service - KronoMata Web Application
	 Loaded: loaded (/etc/systemd/system/KronoMata.Web.service; disabled; vendor preset: enabled)
	 Active: failed (Result: signal) since Fri 2023-08-25 10:01:12 PDT; 55s ago
	Process: 128935 ExecStart=/usr/bin/dotnet /home/bnickel/KronoMata/Publish/Web/KronoMata.Web.dll (code=killed, signal=ABRT)
   Main PID: 128935 (code=killed, signal=ABRT)

The ScheduledJob table in SQLite has a Not Null constraint and that exception is not being handled gracefully.

sudo journalctl -e -u KronoMata.Web.service

Aug 25 10:01:02 flash3r KronoMataWeb[128935]: code = Constraint (19), message = System.Data.SQLite.SQLiteException (0x87AF202F): constraint failed
Aug 25 10:01:02 flash3r KronoMataWeb[128935]: NOT NULL constraint failed: ScheduledJob.HostId
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Data.SQLite.SQLiteDataReader.NextResult()
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Data.Common.DbCommand.ExecuteNonQueryAsync(CancellationToken cancellationToken)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]: --- End of stack trace from previous location ---
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at Dapper.SqlMapper.ExecuteImplAsync(IDbConnection cnn, CommandDefinition command, Object param) in /_/Dapper/SqlMapper.Async.cs:line 655
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at KronoMata.Data.SQLite.SQLiteScheduledJobDataStore.<>c__DisplayClass6_0.<<Update>b__0>d.MoveNext() in D:\Development\KronoMata\KronoMata.Data.SQLite\SQLiteScheduledJobDataS>
Aug 25 10:01:02 flash3r KronoMataWeb[128935]: --- End of stack trace from previous location ---
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Threading.QueueUserWorkItemCallback.Execute()
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Threading.ThreadPoolWorkQueue.Dispatch()
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Aug 25 10:01:12 flash3r systemd[1]: KronoMata.Web.service: Main process exited, code=killed, status=6/ABRT
Aug 25 10:01:12 flash3r systemd[1]: KronoMata.Web.service: Failed with result 'signal'.   

Authentication for Web App

Authentication for the Web App implies the following:

1. Credentials stored in the database.
2. Admin ability to change the credentials
3. Password recovery strategy
4. Splash screen for logging in.
5. Session timeout.
6. SSL requirement

** Intuition is to use bcrypt for password data at rest storage but review this InfoSec Scout article before choosing.

Test Coverage for InMemoryDataStoreProvider

Even though the InMemoryDataStoreProvider uses both the tested SQLiteDataStoreProvider and MockDataStoreProvider, it needs to have test coverage to ensure that it behaves as expected.

The nuget package for KronoMata.Public doesn't copy to output.

This is more of an inconvenience in packaging plugins than a bug but the KronoMata.Public.dll and associated files aren't automatically copied to the output folder when building an IPlugin implementation. If those files are missing from the Package, it will fail to upload.

Create a Docker Image

Create a Docker image that includes the Web App with SQLite and a locally running Agent. PackageRoot and Database paths should be configurable as well as the http port.

Switching a Job's Plugin doesn't redirect for missing configuration

Changing the ScheduledJob Plugin should be checking for the existence of configuration that matches the ScheduledJob and PluginConfiguration for the currently selected Plugin. If none exists, redirect to the Configure view.

Provide Complete and Organized Documentation

Use the Wiki but organize it using a side bar. Documentation should cover installation, usage, and management as well as plugin development.

Required Form Field UI Consistency

Scheduled Job configuration shows an asterisk next to required fields.

No other form provides any indication of required fields but they should. Use the asterisk on all forms.

Boolean not saving correctly for ConfigurationValue

Unchecking a Boolean is not saving the 'False' value or clearing the configuration.

Use Serilog for Logging

Serilog will allow for sending log data to different sinks such as Seq so that KronoMata can be monitored in a centralized and structured way alongside other applications.