Some Zimfarm task are never finishing while the process is obviously finished about zimfarm

[kevinmcmurtrie]

They don't seem to be doing anything.

$ docker logs -t -n 25 ad7dfd16fe54
2024-06-21T20:53:53.607318797Z 	username=pixelmemory
2024-06-21T20:53:53.607327547Z 	webapi_uris=['https://api.farm.openzim.org/v1']
2024-06-21T20:53:53.607335687Z 	workdir=/data
2024-06-21T20:53:53.607350767Z 	task_id=4436db27-adcb-49c4-be0f-d74af0164bf0
2024-06-21T20:53:53.607360177Z [2024-06-21 20:53:53,607: INFO] testing workdir at /data…
2024-06-21T20:53:53.607369237Z [2024-06-21 20:53:53,607: INFO] 	workdir is available and writable
2024-06-21T20:53:53.607377407Z [2024-06-21 20:53:53,607: INFO] testing private key at /etc/ssh/keys/zimfarm…
2024-06-21T20:53:53.691983592Z [2024-06-21 20:53:53,691: INFO] 	private key is available and readable (a3121794f8e27e40c2f20782fae70eab)
2024-06-21T20:53:53.692023802Z [2024-06-21 20:53:53,691: INFO] testing authentication with https://api.farm.openzim.org/v1…
2024-06-21T20:53:54.752378767Z [2024-06-21 20:53:54,752: INFO] 	authentication successful
2024-06-21T20:53:54.752433407Z [2024-06-21 20:53:54,752: INFO] testing docker API on /var/run/docker.sock…
2024-06-21T20:53:54.770756971Z [2024-06-21 20:53:54,770: INFO] 	docker API access successful
2024-06-21T20:53:54.782102019Z [2024-06-21 20:53:54,782: INFO] Host hardware resources:
2024-06-21T20:53:54.782126059Z 	CPU : 20 (total) ;  11 (avail)
2024-06-21T20:53:54.782144659Z 	RAM : 160 GiB (total) ;  134 GiB (avail)
2024-06-21T20:53:54.782151699Z 	Disk: 3 TiB (configured) ; 9.59 TiB (avail) ; 203.13 GiB (reserved) ; 2.8 TiB (remain)
2024-06-21T20:53:54.782161509Z [2024-06-21 20:53:54,782: INFO] registering exit signals
2024-06-21T20:53:54.782169919Z [2024-06-21 20:53:54,782: INFO] Fetching task details for 4436db27-adcb-49c4-be0f-d74af0164bf0
2024-06-21T20:53:55.310394299Z [2024-06-21 20:53:55,310: INFO] Updating task-status=started
2024-06-21T20:53:55.846879537Z [2024-06-21 20:53:55,846: INFO] Setting-up workdir
2024-06-21T20:53:55.849309102Z [2024-06-21 20:53:55,849: INFO] Starting DNS cache
2024-06-21T20:53:57.033792153Z [2024-06-21 20:53:57,033: DEBUG] DNS Cache started using IPs: ['172.17.0.13']
2024-06-21T20:53:57.033822502Z [2024-06-21 20:53:57,033: INFO] Starting scraper. Expects files at: /Yosemite/zimfarm/data/4436db27-adcb-49c4-be0f-d74af0164bf0 
2024-06-21T20:53:57.035737170Z [2024-06-21 20:53:57,035: DEBUG] Pulling image ghcr.io/openzim/ted:dev
2024-06-21T20:54:24.173183862Z [2024-06-21 20:54:24,173: INFO] Updating task-status=scraper_started

$ docker logs -t -n 25 4aed9b3cd658
2024-06-21T10:01:00.779871640Z 	webapi_uris=['https://api.farm.openzim.org/v1']
2024-06-21T10:01:00.779880010Z 	workdir=/data
2024-06-21T10:01:00.779895310Z 	task_id=12c2a80a-f0be-4fed-9d36-1078ab4b440c
2024-06-21T10:01:00.779906280Z [2024-06-21 10:01:00,779: INFO] testing workdir at /data…
2024-06-21T10:01:00.779918120Z [2024-06-21 10:01:00,779: INFO] 	workdir is available and writable
2024-06-21T10:01:00.779926470Z [2024-06-21 10:01:00,779: INFO] testing private key at /etc/ssh/keys/zimfarm…
2024-06-21T10:01:00.863235667Z [2024-06-21 10:01:00,863: INFO] 	private key is available and readable (a3121794f8e27e40c2f20782fae70eab)
2024-06-21T10:01:00.863266247Z [2024-06-21 10:01:00,863: INFO] testing authentication with https://api.farm.openzim.org/v1…
2024-06-21T10:01:01.973726177Z [2024-06-21 10:01:01,973: INFO] 	authentication successful
2024-06-21T10:01:01.973772017Z [2024-06-21 10:01:01,973: INFO] testing docker API on /var/run/docker.sock…
2024-06-21T10:01:01.989067831Z [2024-06-21 10:01:01,989: INFO] 	docker API access successful
2024-06-21T10:01:01.997389094Z [2024-06-21 10:01:01,997: INFO] Host hardware resources:
2024-06-21T10:01:01.997401284Z 	CPU : 20 (total) ;  14 (avail)
2024-06-21T10:01:01.997415134Z 	RAM : 160 GiB (total) ;  148 GiB (avail)
2024-06-21T10:01:01.997420814Z 	Disk: 3 TiB (configured) ; 9.62 TiB (avail) ; 173.13 GiB (reserved) ; 2.83 TiB (remain)
2024-06-21T10:01:01.997427184Z [2024-06-21 10:01:01,997: INFO] registering exit signals
2024-06-21T10:01:01.997447344Z [2024-06-21 10:01:01,997: INFO] Fetching task details for 12c2a80a-f0be-4fed-9d36-1078ab4b440c
2024-06-21T10:01:02.509530841Z [2024-06-21 10:01:02,509: INFO] Updating task-status=started
2024-06-21T10:01:02.997001147Z [2024-06-21 10:01:02,996: INFO] Setting-up workdir
2024-06-21T10:01:02.999647909Z [2024-06-21 10:01:02,999: INFO] Starting DNS cache
2024-06-21T10:01:03.766344035Z [2024-06-21 10:01:03,766: DEBUG] DNS Cache started using IPs: ['172.17.0.10']
2024-06-21T10:01:03.766374615Z [2024-06-21 10:01:03,766: INFO] Starting scraper. Expects files at: /Yosemite/zimfarm/data/12c2a80a-f0be-4fed-9d36-1078ab4b440c 
2024-06-21T10:01:03.767748795Z [2024-06-21 10:01:03,767: DEBUG] Pulling image ghcr.io/openzim/mwoffliner:1.13.0
2024-06-21T10:01:05.176917791Z [2024-06-21 10:01:05,176: INFO] Updating task-status=scraper_started
2024-06-21T19:15:49.220726258Z [2024-06-21 19:15:49,220: ERROR] ConnectionError (attempt 1) for PATCH https://api.farm.openzim.org/v1/tasks/12c2a80a-f0be-4fed-9d36-1078ab4b440c -- HTTPSConnectionPool(host='api.farm.openzim.org', port=443): Read timed out. (read timeout=None)

Sending them a CTRL-C:

$ docker attach ad7dfd16fe54
^C[2024-06-24 06:49:43,950: INFO] received exit signal (Interrupt), shutting down…
[2024-06-24 06:49:43,950: INFO] Stopping all containers and actions
[2024-06-24 06:49:43,950: INFO] Stopping and removing monitor
[2024-06-24 06:49:43,950: INFO] Stopping and removing dnscache
[2024-06-24 06:49:44,314: INFO] Stopping and removing scraper
[2024-06-24 06:49:44,554: INFO] Stopping and removing log_uploader
[2024-06-24 06:49:44,554: INFO] Stopping and removing artifacts_uploader
[2024-06-24 06:49:44,554: INFO] Stopping and removing uploader
[2024-06-24 06:49:44,554: INFO] Stopping and removing checker
[2024-06-24 06:49:44,554: INFO] Removing task workdir /data
[2024-06-24 06:49:44,554: WARNING] ZIM files exists. removing anyway: [('ted_mul_activism_2024-06.zim', '4.66 GiB')]
[2024-06-24 06:49:44,556: INFO] Updating task-status=canceled

$ docker attach 4aed9b3cd658
^C[2024-06-24 06:50:52,176: INFO] received exit signal (Interrupt), shutting down…
[2024-06-24 06:50:52,176: INFO] Stopping all containers and actions
[2024-06-24 06:50:52,176: INFO] Stopping and removing monitor
[2024-06-24 06:50:52,176: INFO] Stopping and removing dnscache
[2024-06-24 06:50:52,530: INFO] Stopping and removing scraper
[2024-06-24 06:50:52,756: INFO] Stopping and removing log_uploader
[2024-06-24 06:50:52,756: INFO] Stopping and removing artifacts_uploader
[2024-06-24 06:50:52,756: INFO] Stopping and removing uploader
[2024-06-24 06:50:52,756: INFO] Stopping and removing checker
[2024-06-24 06:50:52,756: INFO] Removing task workdir /data
[2024-06-24 06:50:52,764: INFO] Updating task-status=canceled

[benoit74]

What we see is that the scraper container have finished (containers f506be0a1d31 and 32424149f271 gone).

But the task containers which are responsible to manage the whole task execution are still there (containers ad7dfd16fe54 and 4aed9b3cd658).

@kevinmcmurtrie could you please give us the last lines of these two containers as well?

[benoit74]

So clearly it looks like we have a concern on the task manager which is not detecting that scraper container has finished / stopped (or not checking anymore maybe ...)

[rgaudin]

Thank you @kevinmcmurtrie ; next time please include the inspect output (at least the State part).

Task manager queries docker API for scraper container status and condiders the following as running:

zimfarm/workers/app/common/docker.py

Line 39 in dc4a00f

RUNNING_STATUSES = ("created", "running", "restarting", "paused", "removing")

• We can exclude created and running and restarting from what was shared. It's possible the containers were paused (external cause, zimfarm doesnt pause containers) or removing (something preventing removal from finishing) though.

I'd wait for another occurrence (with the status clue) before taking action

[benoit74]

Thank you @rgaudin for the wise input!