ddiss / icyci Goto Github PK

I run icyci a12f537 + user service (like #5):

ll|grep icyci >work-before
systemctl --user restart icyci@demo
ll|grep icyci >work-after

diff -wu work-before work-after 
--- work-before	2023-10-04 14:24:35.320689056 +0200
+++ work-after	2023-10-04 14:25:02.789245541 +0200
@@ -24,6 +24,7 @@
 drwx------  2 user users    40 Sep 12 17:41 icyci-workspace1635977155
 drwx------  2 user users    40 Sep  5 11:39 icyci-workspace1637574787
 drwx------  2 user users    40 Sep  5 11:47 icyci-workspace163966524
+drwx------  3 user users    60 Oct  4 14:24 icyci-workspace1672073306
 drwx------  4 user users   140 Sep 26 17:03 icyci-workspace174967818
 drwx------  2 user users    40 Sep 12 17:39 icyci-workspace1757274561
 drwx------  2 user users    40 Sep  5 11:34 icyci-workspace1829529262

Is see that directory shoule be removed. I have zero go-fu, maybe SIGTERM needs to be handled explicitly? So that deferred functions run after main.

Steps to reproduce

• run a test script that takes way too long
• have it timeout
• (re)start icyci service (because the timeout above apparently exits the main loop)
• restarted icyci will fail because of a stale lock

Additional info

Log from the service:

Jun 27 18:37:20 host icyci[80930]: Skipped shutdown
Jun 27 20:36:58 host icyci[80930]: rapido2:/root/build/kselftest/kselftest_install# 2023/06/27 20:36:58 State 5 transition timeout!
- Main PID: 80930 (code=exited, status=1/FAILURE)
- manual restart
Jun 28 15:19:52 host icyci[151323]: 2023/06/28 15:19:52 transitioning from state 0: uninitialized -> 1: clone source repository
Jun 28 15:19:52 host icyci[151331]: Cloning into '/icyci-tmp/icyci-workspace792369250/source'...
Jun 28 15:25:00 host icyci[151323]: 2023/06/28 15:25:00 clone source repository completed successfully
Jun 28 15:25:00 host icyci[151323]: 2023/06/28 15:25:00 transitioning from state 1: clone source repository -> 2: verify branch HEAD
Jun 28 15:25:00 host icyci[151323]: 2023/06/28 15:25:00 GPG verifying commit at origin/icyci-demo tip
Jun 28 15:25:01 host icyci[151323]: 2023/06/28 15:25:00 verify completed successfully
Jun 28 15:25:01 host icyci[151323]: 2023/06/28 15:25:00 transitioning from state 2: verify branch HEAD -> 3: lock commit for testing
Jun 28 15:25:01 host icyci[151585]: From /icyci-linux-results
Jun 28 15:25:01 host icyci[151585]:  * [new ref]                   refs/notes/icyci.locked -> refs/notes/icyci.locked
Jun 28 15:25:01 host icyci[151323]: 2023/06/28 15:25:01 couldn't add git notes lock: exit status 1
Jun 28 15:25:01 host icyci[151323]: 2023/06/28 15:25:01 lock commit for testing failed with exit status 1
Jun 28 15:25:01 host icyci[151323]: 2023/06/28 15:25:01 transitioning from state 3: lock commit for testing -> 8: poll source for new commits
Jun 28 15:25:01 host icyci[151323]: 2023/06/28 15:25:01 Entering poll loop awaiting new icyci-demo commits at 24a3d9bc1b819c695c7c88e1e6e20b4971f010d9

As pointed out by @Werkov in #1 . I'll work on adding much needed test coverage for this workflow in icyci_test.go

Current behavior

Each run of icyci daemon starts with a fresh workspace directory and that includes also the cloned repository. Restart of the daemon means the repository must be full re-cloned and that takes time (and space).

Expected behavior

Only first start of daemon takes long time and repository data can be shared between different runs of icyci daemon.

Considered alternative: git clone --depth=$c but that's rather a workaround, e.g. kernel repo is quite large even without full history.

The set-and-forget goals of icyCI are currently hindered by a lack of resilience against network and server outages.
Most git commands which may contact a remote server already include a retry loop, but the number of retries are hardcoded and there's no back-off.
It might be worth dropping the max-retry count altogether and instead just rely on the user-configurable individual state timeouts.

Reported by Michal at #4 (comment) , but deserves a separate ticket.

The current systemd service file specifies:

ExecStart=/bin/icyci --source-repo ${SOURCE_REPO} --source-branch ${SOURCE_BRANCH} --test-script ${TEST_SCRIPT} --results-repo ${RESULTS_REPO} --push-source-to-results ${PUSH_SOURCE_TO_RESULTS} --poll-interval ${POLL_INTERVAL}

--push-source-to-results is a boolean, which according to the flag package documentation won't accept separate parameters:

-flag=x
-flag x  // non-boolean flags only

Parameters trailing the incorrectly placed ${PUSH_SOURCE_TO_RESULTS} parameter are ignored. Fix for this is to use -flag=x syntax in the systemd service file, and also add error handling if any remaining unprocessed parameters are left over.

We currently call gpgInit() -> gpg --gen-key ... in each test. Doing so is slow and wastes host entropy.
We should be able to do this as part of a global func init() in icyci_test.go. It would likely also make sense to move .gitconfig initialisation into the same function.