caruhome / upparat

🛠️ Update Service for AWS IoT Jobs

Upparat

         __    Hi, I'm Upparat!
 _(\    |@@| /
(__/\__ \--/ __
   \___|----|  |   __
       \ }{ /\ )_ / _\
       /\__/\ \__O (__
      (--/\--)    \__/
      _)(  )(_
     `---''---`

The Upparat is a secure and robust service that runs on your IoT device to download and install files such as firmware updates.

Several hooks provide a seamless integration in your device environment and allow you to use any software update tool such as RAUC, SWUpdate or a custom solution.

Upparat subscribes to AWS Iot Jobs, downloads and verifies the specified file and runs an installation command of your choice. It handles all the nitty gritty details such as cancelled jobs, failed downloads and progress updates.

Installation

pip install upparat

Getting started

• Checkout the examples
• Checkout the documentation

Development

• Create a virtualenv:

  python3 -m venv .venv
  . .venv/bin/activate
  pip3 install --upgrade pip setuptools wheel
  

• Install Upparat in editable mode with development and optional dependencies:

  pip install -e ".[dev,sentry]"
  

Pre-commit hooks

• Install the pre-commit framework.

• Install the pre-commit hooks:

  pre-commit install --install-hooks
  

Unittests & Formatter

docker-compose run test
docker-compose run format

Statemachine

• See visual representation of the internal statemachine.

upparat's Issues

Only HTTP 206 (partial content) should append.

Not really an issue for us right now but we should probably address this:

While looking at the wget sources I noticed that the -c (continue) flag only appends if the Range header was respected and the server response with a 206 (partial content). If the server does not support a range header, the downloaded data will be truncated.

	if (dest
		&& ((config.save_content_on && check_status_code_list(config.save_content_on, resp->code))
		|| (!config.save_content_on
			&& (resp->code == 200 || resp->code == 206 || config.content_on_error)))) {

		// Job re-use?
		xfree(ctx->job->sig_filename);

		ctx->outfd = _prepare_file(resp, dest,
			resp->code == 206 ? O_APPEND : O_TRUNC,  //  <<<<<< THIS <<<<<<
			ctx->job->iri,
			ctx->job->original_url,
			ctx->job->ignore_patterns,
			resp->code == 206 ? ctx->body : NULL,
			ctx->max_memory,
			&ctx->job->sig_filename,
			ctx->job->iri->path);

		if (ctx->outfd == -1)
			ret = -1;
	}

Right now, Uppart does not handle this correctly (always appends). Since we can't be certain that the range header will be accepted, I'd like to address this before we open source it.

Change meaning of force flag to reflect expected behaviour

Right now the force flags meaning is "skip all hooks" (version, pre-download, verify-install).

In hindsight this behaviour is not ideal as it is mostly used to just force an installation if the same version is already on installed, therefore we want to change the meaning of the force flag to only skip the version hook.

Right now, as we use it this leads to a case where the job is marked as good even if the verify-install hook has failed which makes the force flag as we use it unreliable.

fyi @dShapo

[bug] Stops download from every new job when it stopped first job due to URL expiration

Upparat immediately stops download from every new job when it stopped first job due to URL expiration.

• caru-version-0.5.2-2020-01-31-161953-b72e02cdRELEASE-imx6ul-caru-v-1-2-0
• Created a job but set expiration to 5 min instead of 1 hour. URL expired and upparat fetched job again but deleted it because stop event is set (see logs). I deleted the job and created a new one with 1h expiration, but stop event was still set and prevented download. Tried a third time and same happened.
• I restarted upparat manually and it started download successfully

reported by @philippbachmann08

Upparat: Field action in Upparat job document is ignored

Right now, Uppart does not check if a job is meant to be handle by Upparat or not. Therefore any job that gets published will be (tried) to be processed by Upparat. We have planed to use the action field in the job but forgot to add the code that actually checks that...

NOTE: Right now, this is not sever but we need to address this as soon as we plan to add another AWS IoT job handler.

[upparat] Less aggressive backoff behaviour

Problem

Consider limiting the backoff between retires. Right now, Upparat while working can produce the following results with a very shaky but working connection (i.e. mobile). While it slowly downloads the backoff behaviour seems a bit too aggressive in these cases.

Mar 02 13:22:12 159v1292 sh[5489]: [2020-03-02 13:22:12,812] INFO in __init__: State changed from select_job to verify_job.
Mar 02 13:22:12 159v1292 sh[5489]: [2020-03-02 13:22:12,833] INFO in download: Skip download hook: Hook=/etc/upparat/hooks/download.sh, force=True.
Mar 02 13:22:12 159v1292 sh[5489]: [2020-03-02 13:22:12,859] INFO in __init__: State changed from verify_job to download.
Mar 02 13:22:12 159v1292 sh[5489]: [2020-03-02 13:22:12,886] INFO in download: Downloading job to /tmp/upparat/0-5-5-force-159v1292.
Mar 02 13:22:54 159v1292 sh[5489]: [2020-03-02 13:22:54,943] INFO in _common: Backing off download(...) for 0.4s (socket.timeout: The read operation timed out)
Mar 02 13:22:55 159v1292 sh[5489]: [2020-03-02 13:22:55,324] INFO in download: Partial download of 102400 bytes found.
Mar 02 13:22:55 159v1292 sh[5489]: [2020-03-02 13:22:55,336] INFO in download: Downloading job to /tmp/upparat/0-5-5-force-159v1292.
Mar 02 13:23:41 159v1292 sh[5489]: [2020-03-02 13:23:41,547] INFO in _common: Backing off download(...) for 0.6s (socket.timeout: The read operation timed out)
Mar 02 13:23:42 159v1292 sh[5489]: [2020-03-02 13:23:42,126] INFO in download: Partial download of 204800 bytes found.
Mar 02 13:23:42 159v1292 sh[5489]: [2020-03-02 13:23:42,138] INFO in download: Downloading job to /tmp/upparat/0-5-5-force-159v1292.
Mar 02 13:24:26 159v1292 sh[5489]: [2020-03-02 13:24:26,325] INFO in _common: Backing off download(...) for 3.8s (socket.timeout: The read operation timed out)
Mar 02 13:24:30 159v1292 sh[5489]: [2020-03-02 13:24:30,176] INFO in download: Partial download of 307200 bytes found.
Mar 02 13:24:30 159v1292 sh[5489]: [2020-03-02 13:24:30,186] INFO in download: Downloading job to /tmp/upparat/0-5-5-force-159v1292.
Mar 02 13:25:06 159v1292 sh[5489]: [2020-03-02 13:25:06,929] INFO in _common: Backing off download(...) for 6.1s (socket.timeout: The read operation timed out)
Mar 02 13:25:13 159v1292 sh[5489]: [2020-03-02 13:25:13,015] INFO in download: Partial download of 307200 bytes found.
Mar 02 13:25:13 159v1292 sh[5489]: [2020-03-02 13:25:13,025] INFO in download: Downloading job to /tmp/upparat/0-5-5-force-159v1292.
Mar 02 13:25:49 159v1292 sh[5489]: [2020-03-02 13:25:49,233] INFO in _common: Backing off download(...) for 15.7s (socket.timeout: The read operation timed out)
Mar 02 13:26:05 159v1292 sh[5489]: [2020-03-02 13:26:05,001] INFO in download: Partial download of 307200 bytes found.
Mar 02 13:26:05 159v1292 sh[5489]: [2020-03-02 13:26:05,010] INFO in download: Downloading job to /tmp/upparat/0-5-5-force-159v1292.
Mar 02 13:26:46 159v1292 sh[5489]: [2020-03-02 13:26:46,948] INFO in _common: Backing off download(...) for 30.4s (socket.timeout: The read operation timed out)
Mar 02 13:27:17 159v1292 sh[5489]: [2020-03-02 13:27:17,368] INFO in download: Partial download of 409600 bytes found.
Mar 02 13:27:17 159v1292 sh[5489]: [2020-03-02 13:27:17,379] INFO in download: Downloading job to /tmp/upparat/0-5-5-force-159v1292.
Mar 02 13:28:03 159v1292 sh[5489]: [2020-03-02 13:28:03,095] INFO in _common: Backing off download(...) for 0.9s (socket.timeout: The read operation timed out)
Mar 02 13:28:04 159v1292 sh[5489]: [2020-03-02 13:28:04,040] INFO in download: Partial download of 512000 bytes found.
Mar 02 13:28:04 159v1292 sh[5489]: [2020-03-02 13:28:04,053] INFO in download: Downloading job to /tmp/upparat/0-5-5-force-159v1292.
Mar 02 13:28:54 159v1292 sh[5489]: [2020-03-02 13:28:54,073] INFO in _common: Backing off download(...) for 2.4s (socket.timeout: The read operation timed out)
Mar 02 13:28:56 159v1292 sh[5489]: [2020-03-02 13:28:56,483] INFO in download: Partial download of 614400 bytes found.
Mar 02 13:28:56 159v1292 sh[5489]: [2020-03-02 13:28:56,492] INFO in download: Downloading job to /tmp/upparat/0-5-5-force-159v1292.
Mar 02 13:29:38 159v1292 sh[5489]: [2020-03-02 13:29:38,067] INFO in _common: Backing off download(...) for 247.0s (socket.timeout: The read operation timed out)
Mar 02 13:33:45 159v1292 sh[5489]: [2020-03-02 13:33:45,222] INFO in download: Partial download of 716800 bytes found.
Mar 02 13:33:45 159v1292 sh[5489]: [2020-03-02 13:33:45,231] INFO in download: Downloading job to /tmp/upparat/0-5-5-force-159v1292.
Mar 02 13:34:29 159v1292 sh[5489]: [2020-03-02 13:34:29,144] INFO in _common: Backing off download(...) for 192.3s (socket.timeout: The read operation timed out)
Mar 02 13:37:41 159v1292 sh[5489]: [2020-03-02 13:37:41,527] INFO in download: Partial download of 819200 bytes found.
Mar 02 13:37:41 159v1292 sh[5489]: [2020-03-02 13:37:41,546] INFO in download: Downloading job to /tmp/upparat/0-5-5-force-159v1292.
Mar 02 13:38:30 159v1292 sh[5489]: [2020-03-02 13:38:30,166] INFO in _common: Backing off download(...) for 219.6s (socket.timeout: The read operation timed out)
Mar 02 13:42:09 159v1292 sh[5489]: [2020-03-02 13:42:09,875] INFO in download: Partial download of 921600 bytes found.
Mar 02 13:42:09 159v1292 sh[5489]: [2020-03-02 13:42:09,897] INFO in download: Downloading job to /tmp/upparat/0-5-5-force-159v1292.
Mar 02 13:42:48 159v1292 sh[5489]: [2020-03-02 13:42:48,324] INFO in _common: Backing off download(...) for 781.9s (socket.timeout: The read operation timed out)

Potential solutions:

• Provide our own backoff behaviour (the library we use does not allow us to set a max. backoff time, only a max time for the entire execution).

Upparat stuck in an endless loop

Device 8ffghxz1 had several AWS IoT Jobs active, but upparat was stuck in an endless loop, see the log...

Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,455] INFO in __init__: State changed from download to fetch_jobs.                                                                                           
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,526] INFO in select_job: Job execution in progress: 0-6-3-beta-tester                                                                                       
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,536] INFO in __init__: State changed from fetch_jobs to select_job.                                                                                         
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,637] INFO in verify_job: Redo job process                                                                                                                   
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,653] INFO in __init__: State changed from select_job to verify_job.                                                                                         
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,689] INFO in __init__: State changed from verify_job to download.                                                                                           
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,763] INFO in download: Hook successfully completed. Download now allowed.                                                                                   
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,783] INFO in download: Downloading job to /tmp/upparat/0-6-3-beta-tester.                                                                                   
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,952] WARNING in download: URL has expired. Starting over.                                                                                                   
Aug 25 12:32:21 8ffghxz1 sh[219]: [2020-08-25 12:32:21,990] INFO in __init__: State changed from download to fetch_jobs.                                                                                           
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,056] INFO in select_job: Job execution in progress: 0-6-3-beta-tester                                                                                       
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,074] INFO in __init__: State changed from fetch_jobs to select_job.                                                                                         
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,138] INFO in verify_job: Redo job process                                                                                                                   
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,155] INFO in __init__: State changed from select_job to verify_job.                                                                                         
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,182] INFO in __init__: State changed from verify_job to download.                                                                                           
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,276] INFO in download: Hook successfully completed. Download now allowed.                                                                                   
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,305] INFO in download: Downloading job to /tmp/upparat/0-6-3-beta-tester.                                                                                   
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,556] WARNING in download: URL has expired. Starting over.                                                                                                   
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,609] INFO in __init__: State changed from download to fetch_jobs.                                                                                           
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,683] INFO in select_job: Job execution in progress: 0-6-3-beta-tester                                                                                       
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,694] INFO in __init__: State changed from fetch_jobs to select_job.                                                                                         
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,775] INFO in verify_job: Redo job process                                                                                                                   
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,781] INFO in __init__: State changed from select_job to verify_job.                                                                                         
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,810] INFO in __init__: State changed from verify_job to download.                                                                                           
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,883] INFO in download: Hook successfully completed. Download now allowed.
Aug 25 12:32:22 8ffghxz1 sh[219]: [2020-08-25 12:32:22,923] INFO in download: Downloading job to /tmp/upparat/0-6-3-beta-tester.
Aug 25 12:32:23 8ffghxz1 sh[219]: [2020-08-25 12:32:23,054] WARNING in download: URL has expired. Starting over.

Canceling older Jobs in cloud enabled device to fetch and execute the latest one
Bundle version: caru-version-0.5.6-2020-04-16-142853-456dec37RELEASE-imx6ul-caru-v-1-2-0
Upparat version: 1.4 - git commit hash = a5beb33b42119692091cb9d80140284d91326e9f

Upparat: download_location in /tmp/ might be cleaned before download

See Sentry

Explanation:

It's possible that the system might clean up /tmp/upparat before download.

Fix:

• Move folder creation to download state instead of config (which does this at service startup).

Upparat stuck in download state

Upparat gets stuck in download state on ConnectionResetError:

Jul 01 08:32:40 inc7qj59 systemd[1]: Started Upparat AWS IoT file installer.
Jul 01 08:32:59 inc7qj59 sh[248]: Loading config from file: /etc/upparat/upparat.conf
Jul 01 08:32:59 inc7qj59 sh[248]: [2019-07-01 08:32:59,915] WARNING in mqtt: Unable to subscribe to topic $aws/things/inc7qj59/jobs/get/+: The client is not currently connected.
Jul 01 08:32:59 inc7qj59 sh[248]: [2019-07-01 08:32:59,991] INFO in mqtt: Connection Accepted.
Oct 20 14:08:16 inc7qj59 sh[248]: [2020-10-20 14:08:16,725] INFO in select_job: Job execution in progress: CO2_201020
Oct 20 14:08:16 inc7qj59 sh[248]: [2020-10-20 14:08:16,736] INFO in __init__: State changed from fetch_jobs to select_job.
Oct 20 14:08:16 inc7qj59 sh[248]: [2020-10-20 14:08:16,797] INFO in verify_job: Installation done
Oct 20 14:08:16 inc7qj59 sh[248]: [2020-10-20 14:08:16,805] INFO in __init__: State changed from select_job to verify_job.
Oct 20 14:08:16 inc7qj59 sh[248]: [2020-10-20 14:08:16,820] INFO in verify_installation: Skip version check
Oct 20 14:08:16 inc7qj59 sh[248]: [2020-10-20 14:08:16,828] INFO in __init__: State changed from verify_job to verify_installation.
Oct 20 14:08:16 inc7qj59 sh[248]: [2020-10-20 14:08:16,837] INFO in __init__: State changed from verify_installation to fetch_jobs.
Oct 20 14:08:17 inc7qj59 sh[248]: [2020-10-20 14:08:16,994] INFO in __init__: State changed from fetch_jobs to monitor.
Oct 20 14:18:57 inc7qj59 sh[248]: [2020-10-20 14:18:57,148] INFO in select_job: Start queued job execution: CO2_201020-2
Oct 20 14:18:57 inc7qj59 sh[248]: [2020-10-20 14:18:57,162] INFO in __init__: State changed from monitor to select_job.
Oct 20 14:18:57 inc7qj59 sh[248]: [2020-10-20 14:18:57,326] INFO in verify_job: Skip version check
Oct 20 14:18:57 inc7qj59 sh[248]: [2020-10-20 14:18:57,335] INFO in __init__: State changed from select_job to verify_job.
Oct 20 14:18:57 inc7qj59 sh[248]: [2020-10-20 14:18:57,348] INFO in download: Skip download hook: Hook=/etc/upparat/hooks/download.sh, force=yes.
Oct 20 14:18:57 inc7qj59 sh[248]: [2020-10-20 14:18:57,360] INFO in __init__: State changed from verify_job to download.
Oct 20 14:18:57 inc7qj59 sh[248]: [2020-10-20 14:18:57,373] INFO in download: Downloading job to /tmp/upparat/CO2_201020-2.
Oct 20 14:26:48 inc7qj59 sh[248]: Exception in thread Thread-2:
Oct 20 14:26:48 inc7qj59 sh[248]: Traceback (most recent call last):
Oct 20 14:26:48 inc7qj59 sh[248]:   File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
Oct 20 14:26:48 inc7qj59 sh[248]:     self.run()
Oct 20 14:26:48 inc7qj59 sh[248]:   File "/usr/lib/python3.7/threading.py", line 865, in run
Oct 20 14:26:48 inc7qj59 sh[248]:     self._target(*self._args, **self._kwargs)
Oct 20 14:26:48 inc7qj59 sh[248]:   File "/usr/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry
Oct 20 14:26:48 inc7qj59 sh[248]:     ret = target(*args, **kwargs)
Oct 20 14:26:48 inc7qj59 sh[248]:   File "/usr/lib/python3.7/site-packages/upparat/statemachine/download.py", line 72, in download
Oct 20 14:26:48 inc7qj59 sh[248]:     data = source.read(READ_CHUNK_SIZE_BYTES)
Oct 20 14:26:48 inc7qj59 sh[248]:   File "/usr/lib/python3.7/http/client.py", line 447, in read
Oct 20 14:26:48 inc7qj59 sh[248]:     n = self.readinto(b)
Oct 20 14:26:48 inc7qj59 sh[248]:   File "/usr/lib/python3.7/http/client.py", line 491, in readinto
Oct 20 14:26:48 inc7qj59 sh[248]:     n = self.fp.readinto(b)
Oct 20 14:26:48 inc7qj59 sh[248]:   File "/usr/lib/python3.7/socket.py", line 589, in readinto
Oct 20 14:26:48 inc7qj59 sh[248]:     return self._sock.recv_into(b)
Oct 20 14:26:48 inc7qj59 sh[248]:   File "/usr/lib/python3.7/ssl.py", line 1052, in recv_into
Oct 20 14:26:48 inc7qj59 sh[248]:     return self.read(nbytes, buffer)
Oct 20 14:26:48 inc7qj59 sh[248]:   File "/usr/lib/python3.7/ssl.py", line 911, in read
Oct 20 14:26:48 inc7qj59 sh[248]:     return self._sslobj.read(len, buffer)
Oct 20 14:26:48 inc7qj59 sh[248]: ConnectionResetError: [Errno 104] Connection reset by peer