Comments (5)
They don't seem to be doing anything.
$ docker logs -t -n 25 ad7dfd16fe54
2024-06-21T20:53:53.607318797Z username=pixelmemory
2024-06-21T20:53:53.607327547Z webapi_uris=['https://api.farm.openzim.org/v1']
2024-06-21T20:53:53.607335687Z workdir=/data
2024-06-21T20:53:53.607350767Z task_id=4436db27-adcb-49c4-be0f-d74af0164bf0
2024-06-21T20:53:53.607360177Z [2024-06-21 20:53:53,607: INFO] testing workdir at /data…
2024-06-21T20:53:53.607369237Z [2024-06-21 20:53:53,607: INFO] workdir is available and writable
2024-06-21T20:53:53.607377407Z [2024-06-21 20:53:53,607: INFO] testing private key at /etc/ssh/keys/zimfarm…
2024-06-21T20:53:53.691983592Z [2024-06-21 20:53:53,691: INFO] private key is available and readable (a3121794f8e27e40c2f20782fae70eab)
2024-06-21T20:53:53.692023802Z [2024-06-21 20:53:53,691: INFO] testing authentication with https://api.farm.openzim.org/v1…
2024-06-21T20:53:54.752378767Z [2024-06-21 20:53:54,752: INFO] authentication successful
2024-06-21T20:53:54.752433407Z [2024-06-21 20:53:54,752: INFO] testing docker API on /var/run/docker.sock…
2024-06-21T20:53:54.770756971Z [2024-06-21 20:53:54,770: INFO] docker API access successful
2024-06-21T20:53:54.782102019Z [2024-06-21 20:53:54,782: INFO] Host hardware resources:
2024-06-21T20:53:54.782126059Z CPU : 20 (total) ; 11 (avail)
2024-06-21T20:53:54.782144659Z RAM : 160 GiB (total) ; 134 GiB (avail)
2024-06-21T20:53:54.782151699Z Disk: 3 TiB (configured) ; 9.59 TiB (avail) ; 203.13 GiB (reserved) ; 2.8 TiB (remain)
2024-06-21T20:53:54.782161509Z [2024-06-21 20:53:54,782: INFO] registering exit signals
2024-06-21T20:53:54.782169919Z [2024-06-21 20:53:54,782: INFO] Fetching task details for 4436db27-adcb-49c4-be0f-d74af0164bf0
2024-06-21T20:53:55.310394299Z [2024-06-21 20:53:55,310: INFO] Updating task-status=started
2024-06-21T20:53:55.846879537Z [2024-06-21 20:53:55,846: INFO] Setting-up workdir
2024-06-21T20:53:55.849309102Z [2024-06-21 20:53:55,849: INFO] Starting DNS cache
2024-06-21T20:53:57.033792153Z [2024-06-21 20:53:57,033: DEBUG] DNS Cache started using IPs: ['172.17.0.13']
2024-06-21T20:53:57.033822502Z [2024-06-21 20:53:57,033: INFO] Starting scraper. Expects files at: /Yosemite/zimfarm/data/4436db27-adcb-49c4-be0f-d74af0164bf0
2024-06-21T20:53:57.035737170Z [2024-06-21 20:53:57,035: DEBUG] Pulling image ghcr.io/openzim/ted:dev
2024-06-21T20:54:24.173183862Z [2024-06-21 20:54:24,173: INFO] Updating task-status=scraper_started
$ docker logs -t -n 25 4aed9b3cd658
2024-06-21T10:01:00.779871640Z webapi_uris=['https://api.farm.openzim.org/v1']
2024-06-21T10:01:00.779880010Z workdir=/data
2024-06-21T10:01:00.779895310Z task_id=12c2a80a-f0be-4fed-9d36-1078ab4b440c
2024-06-21T10:01:00.779906280Z [2024-06-21 10:01:00,779: INFO] testing workdir at /data…
2024-06-21T10:01:00.779918120Z [2024-06-21 10:01:00,779: INFO] workdir is available and writable
2024-06-21T10:01:00.779926470Z [2024-06-21 10:01:00,779: INFO] testing private key at /etc/ssh/keys/zimfarm…
2024-06-21T10:01:00.863235667Z [2024-06-21 10:01:00,863: INFO] private key is available and readable (a3121794f8e27e40c2f20782fae70eab)
2024-06-21T10:01:00.863266247Z [2024-06-21 10:01:00,863: INFO] testing authentication with https://api.farm.openzim.org/v1…
2024-06-21T10:01:01.973726177Z [2024-06-21 10:01:01,973: INFO] authentication successful
2024-06-21T10:01:01.973772017Z [2024-06-21 10:01:01,973: INFO] testing docker API on /var/run/docker.sock…
2024-06-21T10:01:01.989067831Z [2024-06-21 10:01:01,989: INFO] docker API access successful
2024-06-21T10:01:01.997389094Z [2024-06-21 10:01:01,997: INFO] Host hardware resources:
2024-06-21T10:01:01.997401284Z CPU : 20 (total) ; 14 (avail)
2024-06-21T10:01:01.997415134Z RAM : 160 GiB (total) ; 148 GiB (avail)
2024-06-21T10:01:01.997420814Z Disk: 3 TiB (configured) ; 9.62 TiB (avail) ; 173.13 GiB (reserved) ; 2.83 TiB (remain)
2024-06-21T10:01:01.997427184Z [2024-06-21 10:01:01,997: INFO] registering exit signals
2024-06-21T10:01:01.997447344Z [2024-06-21 10:01:01,997: INFO] Fetching task details for 12c2a80a-f0be-4fed-9d36-1078ab4b440c
2024-06-21T10:01:02.509530841Z [2024-06-21 10:01:02,509: INFO] Updating task-status=started
2024-06-21T10:01:02.997001147Z [2024-06-21 10:01:02,996: INFO] Setting-up workdir
2024-06-21T10:01:02.999647909Z [2024-06-21 10:01:02,999: INFO] Starting DNS cache
2024-06-21T10:01:03.766344035Z [2024-06-21 10:01:03,766: DEBUG] DNS Cache started using IPs: ['172.17.0.10']
2024-06-21T10:01:03.766374615Z [2024-06-21 10:01:03,766: INFO] Starting scraper. Expects files at: /Yosemite/zimfarm/data/12c2a80a-f0be-4fed-9d36-1078ab4b440c
2024-06-21T10:01:03.767748795Z [2024-06-21 10:01:03,767: DEBUG] Pulling image ghcr.io/openzim/mwoffliner:1.13.0
2024-06-21T10:01:05.176917791Z [2024-06-21 10:01:05,176: INFO] Updating task-status=scraper_started
2024-06-21T19:15:49.220726258Z [2024-06-21 19:15:49,220: ERROR] ConnectionError (attempt 1) for PATCH https://api.farm.openzim.org/v1/tasks/12c2a80a-f0be-4fed-9d36-1078ab4b440c -- HTTPSConnectionPool(host='api.farm.openzim.org', port=443): Read timed out. (read timeout=None)
Sending them a CTRL-C:
$ docker attach ad7dfd16fe54
^C[2024-06-24 06:49:43,950: INFO] received exit signal (Interrupt), shutting down…
[2024-06-24 06:49:43,950: INFO] Stopping all containers and actions
[2024-06-24 06:49:43,950: INFO] Stopping and removing monitor
[2024-06-24 06:49:43,950: INFO] Stopping and removing dnscache
[2024-06-24 06:49:44,314: INFO] Stopping and removing scraper
[2024-06-24 06:49:44,554: INFO] Stopping and removing log_uploader
[2024-06-24 06:49:44,554: INFO] Stopping and removing artifacts_uploader
[2024-06-24 06:49:44,554: INFO] Stopping and removing uploader
[2024-06-24 06:49:44,554: INFO] Stopping and removing checker
[2024-06-24 06:49:44,554: INFO] Removing task workdir /data
[2024-06-24 06:49:44,554: WARNING] ZIM files exists. removing anyway: [('ted_mul_activism_2024-06.zim', '4.66 GiB')]
[2024-06-24 06:49:44,556: INFO] Updating task-status=canceled
$ docker attach 4aed9b3cd658
^C[2024-06-24 06:50:52,176: INFO] received exit signal (Interrupt), shutting down…
[2024-06-24 06:50:52,176: INFO] Stopping all containers and actions
[2024-06-24 06:50:52,176: INFO] Stopping and removing monitor
[2024-06-24 06:50:52,176: INFO] Stopping and removing dnscache
[2024-06-24 06:50:52,530: INFO] Stopping and removing scraper
[2024-06-24 06:50:52,756: INFO] Stopping and removing log_uploader
[2024-06-24 06:50:52,756: INFO] Stopping and removing artifacts_uploader
[2024-06-24 06:50:52,756: INFO] Stopping and removing uploader
[2024-06-24 06:50:52,756: INFO] Stopping and removing checker
[2024-06-24 06:50:52,756: INFO] Removing task workdir /data
[2024-06-24 06:50:52,764: INFO] Updating task-status=canceled
from zimfarm.
What we see is that the scraper container have finished (containers f506be0a1d31
and 32424149f271
gone).
But the task containers which are responsible to manage the whole task execution are still there (containers ad7dfd16fe54
and 4aed9b3cd658
).
@kevinmcmurtrie could you please give us the last lines of these two containers as well?
from zimfarm.
So clearly it looks like we have a concern on the task manager which is not detecting that scraper container has finished / stopped (or not checking anymore maybe ...)
from zimfarm.
Thank you @kevinmcmurtrie ; next time please include the inspect
output (at least the State
part).
Task manager queries docker API for scraper container status and condiders the following as running:
zimfarm/workers/app/common/docker.py
Line 39 in dc4a00f
- We can exclude
created
andrunning
andrestarting
from what was shared. It's possible the containers werepaused
(external cause, zimfarm doesnt pause containers) orremoving
(something preventing removal from finishing) though.
I'd wait for another occurrence (with the status clue) before taking action
from zimfarm.
Thank you @rgaudin for the wise input!
from zimfarm.
Related Issues (20)
- Cancelling a task is not resilient enough HOT 3
- Worker downtime is wrong
- Zim files created in dev should default to manual refresh HOT 2
- Relaxed filename is too relaxed
- Fix sotoki scraper configuration
- Watcher is not detecting updated files when SE is fixing issues HOT 3
- Upload logs and artifacts even when task is cancelled
- Name resolution errors in StackExchange watcher
- Configuration of Zimfarm local instance HOT 1
- Installing monitoring of a task is failing HOT 2
- Workers: report real disk size (and CPU / RAM) HOT 7
- StackExchanger watcher regularly restarts after connection reset by peer errors HOT 1
- Is the worker incorrectly checking the space already consumed by the Zimfarm? HOT 8
- Collect zimfarm usage statistics
- Adapt Zimfarm to MWoffliner 1.14 HOT 3
- Ensure we report MiB/GiB/... values everywhere
- Migrate to MWoffliner 1.14 HOT 6
- Recipe edit form consider there is always a pending modification
- Remove all control characters in task properties
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zimfarm.