mufaka / kronomata Goto Github PK

When an agent is running several jobs at the same time, it can cause a database locking issue because of the simultaneous requests to write the JobHistory (via the API call).

07:53:13 fail: KronoMata.Web.Controllers.AgentController[0] 
=> SpanId:64d28d8bfe22583a, TraceId:2af3454693e0c2e76487aeb5c1482021, ParentId:0000000000000000 => ConnectionId:0HMT8JPAB6OQS 
=> RequestPath:/api/Agent/history RequestId:0HMT8JPAB6OQS:00000007 
=> KronoMata.Web.Controllers.AgentController.Post (KronoMata.Web) Error creating JobHistory code = Busy (5), 
message = System.Data.SQLite.SQLiteException (0x87AF00AA): database is locked database is locked    
at System.Data.SQLite.SQLite3.Prepare(SQLiteConnection cnn, SQLiteCommand command, String strSql, SQLiteStatement previous, UInt32 timeoutMS, String& strRemain)    
at System.Data.SQLite.SQLiteCommand.BuildNextCommand()    
at System.Data.SQLite.SQLiteDataReader.NextResult()    
at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)    
at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)    
at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)    
at System.Data.SQLite.SQLiteConnection.Open()    
at KronoMata.Data.DbConnectionDataStoreBase.Execute(Action`1 action) in D:\Development\KronoMata\KronoMata.Data\DbConnectionDataStoreBase.cs:line 41    
at KronoMata.Data.SQLite.SQLiteJobHistoryDataStore.Create(JobHistory jobHistory) in D:\Development\KronoMata\KronoMata.Data.SQLite\SQLiteJobHistoryDataStore.cs:line 46    
at KronoMata.Data.InMemory.InMemoryJobHistoryDataStore.Create(JobHistory jobHistory) in D:\Development\KronoMata\KronoMata.Data.InMemory\InMemoryJobHistoryDataStore.cs:line 19    
at KronoMata.Web.Controllers.AgentController.Post(JobHistory history) in D:\Development\KronoMata\KronoMata.Web\Controllers\AgentController.cs:line 93

This does cause a JobHistory save to be missed the following screen capture shows only 6 history rows create for 7:53. It is expected that there are 7.

A Scheduled Job with 'All' selected for Host doesn't appear to work. No jobs are run and when viewing the Host, the Scheduled Job doesn't appear.

It would be nice to know which hosts are running the Scheduled Jobs. For Hosts you can tell what Scheduled Jobs they are running but not vice versa.

Unchecking a Boolean is not saving the 'False' value or clearing the configuration.

This is more of an inconvenience in packaging plugins than a bug but the KronoMata.Public.dll and associated files aren't automatically copied to the output folder when building an IPlugin implementation. If those files are missing from the Package, it will fail to upload.

Create a Docker image that includes the Web App with SQLite and a locally running Agent. PackageRoot and Database paths should be configurable as well as the http port.

Job History shows an 'End of Central Directory record could not be found.' error message when the package zip file cannot be extracted.

The zip file is corrupted when serving from a linux KronoMata.Web instance to a Windows KronoMata.Agent instance. The same zip file is not corrupt when the agent is running on linux.

Agents download packages in order to run plugins. If a package is deleted from the web application, it should also be deleted from the agents local store. Before implementing, brainstorm other use cases where this plugin can be used.

1. Call a new API method for getting a list of currently defined packages.
2. Delete local files that are not included in that list.

** Consider pros/cons of using the ScheduledJob list to do this. If packages are local and there are no current scheduled jobs for them, delete?

#10 Is a prerequisite.

1. Add an API endpoint for expiring Job History. Use global settings for the expiration parameters.
2. Create an IPlugin that calls the API endpoint.

The Agent ticks every minute to check for jobs to run. If a job runs for more than a minute (eg: backup job) then ticks will be missed until that job finishes.

This needs to be thought out a bit more but the gist is as follows:

1. Create another table named RunNowJob that mimics ScheduledJob but also includes a flag for completed.
2. Create another table name RunNowHistory that mimics JobHistory but links to RunNowJob
3. RunNowJob should allow for selecting hosts in the same way ScheduledJob does.
4. A separate RunNowJob row should be created for each selected host.
5. UNION RunNowJob and ScheduledJob for API call that Agent uses to get a list of jobs.
6. UNION RunNowHistory and JobHistory results
7. Respect new tables when expiring history

** It think that RunNowJobs should have a default EndTime to prevent them from running repeatedly as well as defaulting to Frequency Minute, Interval 1.

Brainstorm for an easier way to implement this. I think adding a flag to ScheduledJob for RunNow might work as long as it ends up creating a ScheduledJob for each selected host. UI can enforce defaults when RunNow is selected as well as separating the display of ScheduledJobs into groups. If we do go this route, it's probably better to add a ScheduleType (Recurrence, RunNow, RunOnce).

This should be an IDataStoreProvider implementation.

1. The implementation should have a "backing data store" that persists data.
2. Caching should only be done for relatively static data (not JobHistory)
3. Implementation should work as a write-through cache.

Update the Web application to use this store.

The agent service exits immediately when running as a systemd service on linux. The agent runs fine if started manually.

Service definition in /etc/systemd/system/KronoMata.Agent.service

[Unit]
Description=KronoMata Agent Application

[Service]
WorkingDirectory=/home/bnickel/KronoMata/Publish/Agent
ExecStart=/usr/bin/dotnet /home/bnickel/KronoMata/Publish/Agent/KronoMata.Agent.dll
SyslogIdentifier=KronoMataAgent

[Install]
WantedBy=multi-user.target

Starting service and checking status shows an inactive (dead) service.

bnickel@flash3r:/etc/systemd/system$ sudo systemctl start KronoMata.Agent
bnickel@flash3r:/etc/systemd/system$ sudo systemctl status KronoMata.Agent
● KronoMata.Agent.service - KronoMata Agent Application
	 Loaded: loaded (/etc/systemd/system/KronoMata.Agent.service; disabled; vendor preset: enabled)
	 Active: inactive (dead)

The full log shows it starting but then stopping.

sudo journalctl -u KronoMata.Agent.service	 
Aug 25 09:50:08 flash3r systemd[1]: Started KronoMata Agent Application.
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 dbug: Microsoft.Extensions.Hosting.Internal.Host[1] Hosting starting
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: KronoMata.Agent.PluginRunner[0] KronoMata Agent starting.
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 dbug: Microsoft.Extensions.Hosting.Internal.Host[3] Hosting stopping
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: Microsoft.Hosting.Lifetime[0] Application is shutting down...
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down.
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: Microsoft.Hosting.Lifetime[0] Hosting environment: Production
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: Microsoft.Hosting.Lifetime[0] Content root path: /home/bnickel/KronoMata/Publish/Agent/
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 dbug: Microsoft.Extensions.Hosting.Internal.Host[2] Hosting started
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 dbug: Microsoft.Extensions.Hosting.Internal.Host[3] Hosting stopping
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: KronoMata.Agent.PluginRunner[0] KronoMata Agent stopped.
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 dbug: Microsoft.Extensions.Hosting.Internal.Host[4] Hosting stopped
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 info: KronoMata.Agent.PluginRunner[0] KronoMata Agent stopped.
Aug 25 09:50:09 flash3r KronoMataAgent[130471]: 09:50:09 dbug: Microsoft.Extensions.Hosting.Internal.Host[4] Hosting stopped
Aug 25 09:50:09 flash3r systemd[1]: KronoMata.Agent.service: Succeeded.	 

Use the Wiki but organize it using a side bar. Documentation should cover installation, usage, and management as well as plugin development.

This happened when updating a ScheduledJob to run on 'All' hosts.

sudo systemctl status KronoMata.Web

● KronoMata.Web.service - KronoMata Web Application
	 Loaded: loaded (/etc/systemd/system/KronoMata.Web.service; disabled; vendor preset: enabled)
	 Active: failed (Result: signal) since Fri 2023-08-25 10:01:12 PDT; 55s ago
	Process: 128935 ExecStart=/usr/bin/dotnet /home/bnickel/KronoMata/Publish/Web/KronoMata.Web.dll (code=killed, signal=ABRT)
   Main PID: 128935 (code=killed, signal=ABRT)

The ScheduledJob table in SQLite has a Not Null constraint and that exception is not being handled gracefully.

sudo journalctl -e -u KronoMata.Web.service

Aug 25 10:01:02 flash3r KronoMataWeb[128935]: code = Constraint (19), message = System.Data.SQLite.SQLiteException (0x87AF202F): constraint failed
Aug 25 10:01:02 flash3r KronoMataWeb[128935]: NOT NULL constraint failed: ScheduledJob.HostId
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Data.SQLite.SQLiteDataReader.NextResult()
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Data.Common.DbCommand.ExecuteNonQueryAsync(CancellationToken cancellationToken)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]: --- End of stack trace from previous location ---
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at Dapper.SqlMapper.ExecuteImplAsync(IDbConnection cnn, CommandDefinition command, Object param) in /_/Dapper/SqlMapper.Async.cs:line 655
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at KronoMata.Data.SQLite.SQLiteScheduledJobDataStore.<>c__DisplayClass6_0.<<Update>b__0>d.MoveNext() in D:\Development\KronoMata\KronoMata.Data.SQLite\SQLiteScheduledJobDataS>
Aug 25 10:01:02 flash3r KronoMataWeb[128935]: --- End of stack trace from previous location ---
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Threading.QueueUserWorkItemCallback.Execute()
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Threading.ThreadPoolWorkQueue.Dispatch()
Aug 25 10:01:02 flash3r KronoMataWeb[128935]:    at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Aug 25 10:01:12 flash3r systemd[1]: KronoMata.Web.service: Main process exited, code=killed, status=6/ABRT
Aug 25 10:01:12 flash3r systemd[1]: KronoMata.Web.service: Failed with result 'signal'.   

Serilog will allow for sending log data to different sinks such as Seq so that KronoMata can be monitored in a centralized and structured way alongside other applications.

If the PackageRoot directory doesn't exist, the Agent cannot download and extract plugins. By default, the PackageRoot is configured as a subdirectory of the agent so it should be creatable.

If a release does not contain the empty PackageRoot sub directory, an error similar to the following will occur.

One or more errors occurred. (Could not find a part of the path 'path to \PackageRoot\92bee86d-58a1-48e2-b9b8-99d3d3adf179.zip'.)

After a period of prolonged running (unsure how long) unzipping package files fails for the Agent and the Web App. Restarting solves the issue for a period of time.

This appears to be specific to Ubuntu 22.04 with the following dotnet versions: (dotnet --info)
Microsoft.AspNetCore.App 6.0.22
Microsoft.NETCore.App 6.0.22

Ubuntu 20.04 with the 6.0.21 dotnet versions doesn't exhibit this behavior nor does Windows.

Changing the ScheduledJob Plugin should be checking for the existence of configuration that matches the ScheduledJob and PluginConfiguration for the currently selected Plugin. If none exists, redirect to the Configure view.

The KronoMata.Agent application should have a means to verify that the code being run is what the KronoMata.Web application expects. Right now it would be fairly easy to decompile a plugin, alter it in a malicious way, and then replace the existing one.

One option is to store a checksum on PluginMetaData and have the Agent check that before executing the plugin.

1. Provide a means to purge Job History based on age and/or log size. IJobHistoryDataStore should provide a method for deleting Job History that accepts parameters for daysToKeep and maxHistoryRecords.

2. Global Configuration should have these two parameters defined with some reasonable defaults (14 days, 2500 items)

3. Add a section to the Settings view for manually running expiration.

Only do this on the History view. This should be configurable on the grid and default to off, refreshing every 30 seconds.

Implement a set timeout function for refreshing the jsGrid at a 30 second interval. Add an icon in the card header for enabling / disabling this.

Right now you can either schedule a job to run on a single host or all of them.

1. Change the definition of ScheduledJob.HostId to a string variant data type. (rename to Hosts)
2. Allow for selecting multiple hosts when saving a ScheduledJob; save as comma separated list.
3. Update ScheduledJob listing by machine name to respect the list of hosts on ScheduledJob.

Allow for deletion of Package, PluginMetaData, Host, and ScheduledJob. These are potentially cascading deletes but required in order to effectively manage the application.

Authentication for the Web App implies the following:

1. Credentials stored in the database.
2. Admin ability to change the credentials
3. Password recovery strategy
4. Splash screen for logging in.
5. Session timeout.
6. SSL requirement

** Intuition is to use bcrypt for password data at rest storage but review this InfoSec Scout article before choosing.

The API root should contain a version (eg: /api/v1/) in order to support the ability to enhance the API without introducing breaking changes for existing agents.

GlobalConfiguration and ConfigurationValues are stored in plain text. IsMasked and DataType.Password are only considered on the front end for hiding the values but they should be encrypted in the database.

Even though the InMemoryDataStoreProvider uses both the tested SQLiteDataStoreProvider and MockDataStoreProvider, it needs to have test coverage to ensure that it behaves as expected.

Update the Packages view to use col-12 for card widths to match the style of the rest of the views.

The test cases for SQLite, in Test.KronoMata.Data.SQLite, were cut and paste from the tests written in Test.KronoMata.Data.Mock. Additional test cases were added for SQLite but not for the Mock implementation.

The IDataStoreProvider and supporting IDataStoreXxx interfaces define the expected behavior of implementations and those are what should be tested against.

Refactor the tests in a manner that allows for testing implementations of IDataStoreProvider with minimal code duplication. Use the SQLite tests as the template and then refactor Test.KronoMata.Data.SQLite and Test.KronoMata.Data.Mock to use the common tests.

Take the following test case that looks identical in both sets of tests:

public void Can_Create()
{
	var now = DateTime.Now;

	var scheduledJob = new ScheduledJob()
	{
		PluginMetaDataId = 1,
		HostId = 1,
		Name = "Name",
		Description = "Description",
		Frequency = ScheduleFrequency.Week,
		Interval = 2,
		StartTime = now,
		EndTime = now,
		IsEnabled = true,
		InsertDate = now,
		UpdateDate = now
	};

	_provider.ScheduledJobDataStore.Create(scheduledJob);

	Assert.That(scheduledJob.Id, Is.EqualTo(1));
}

There should be an elegant way to just pass the _provider to a shared template like the following:

public void Can_Create()
{
	var scheduledJob = SomeCommonScheduledJobCan_CreateMethod(_provider);
	Assert.That(scheduledJob.Id, Is.EqualTo(1));
}

A job that was scheduled to run daily with specified didn't run at the correct time on days after the specified start date and time.

It was expected that the job would have run at 3:05, 3:10, 3:15, 3:20 and at those same minutes for the 6 o'clock hour as well.

Job History RunTime and CompletionTime needs to be stored as UTC and then converted for display in the web app.

ScheduledJob dates will need to be thought out more before implementing because there might be a case where agents span multiple timezones and you want them to run at the same UTC time. But there are other use cases where you want the job to run at specific local times regardless of timezone (eg: fetch files from SFTP at 3:00 am).

Scheduled Job configuration shows an asterisk next to required fields.

No other form provides any indication of required fields but they should. Use the asterisk on all forms.

All API endpoints need to be protected with a key and any controller method that returns data should require authorization.