Comments (7)
I like that worker admins can set resources for zimfarm and don't have to entirely donate all host resources to the worker. Was done on purpose but can be reassessed indeed.
Probably true for RAM / CPU as well, this would help diagnose problems
I don't see how that would help TBH.
Also, we should keep in mind that there's a major difference between CPU/RAM which are enforced by docker and disk which is not at all.
from zimfarm.
I like that worker admins can set resources for zimfarm and don't have to entirely donate all host resources to the worker.
I like it too (even I would recommend to do it with VMs so that isolation is more realist, but it is not a requirement, we must support both scenarii)
However here issue was opposite: disk declared by owner was 1TB (whole disk size) while in fact only 30G was available in /var
due to advanced partitioning scheme (worker owner expected /var
to contain only logs). It made it hard to realize that disk full errors where real and not a bug.
from zimfarm.
Is the doc misleading or was it overlooked?
from zimfarm.
When I read the doc, I understand that Zimfarm data will be stored in ZIMFARM_ROOT
, but this is wrong. Or at least zimit scraper / zimit configuration is not respecting a Zimfarm constraints (I've never heard of before, or at least forgotten) that every big stuff must be pushed in /output
. It looks like only /output
(inside the container) in mounted to {ZIMFARM_ROOT}/data/{uuid}
. Everything else is left in /var
, not only logs. And zimit is pushing its warc to /crawls
subfolder.
from zimfarm.
Everything else is left in /var, not only logs.
Yes, everything else is not managed and thus handled by docker. Those container data as well as image data are the reason we recommend to change the default folder. We should not mention logs there as it's misleading IMO.
And zimit is pushing its warc to /crawls subfolder.
That's a zimit bug that must be opened.
from zimfarm.
We should not mention logs there as it's misleading IMO.
Yes, this is what misled me at least
That's a zimit bug that must be opened.
Double checking, it is not indeed. I.e. by default zimit creates WARCs in a temporary subfolder of /output, see --cwd
below
[zimit::2024-05-27 08:09:51,768] INFO:Running browsertrix-crawler crawl: crawl --failOnFailedSeed --waitUntil load --title Zimit test website --description A test website for Zimit --depth 0 --timeout 90 --lang eng --behaviors autoplay,autofetch,siteSpecific --behaviorTimeout 90 --diskUtilization 90 --url https://tmp.kiwix.org/ci/test-website/ --userAgentSuffix +Zimit [email protected] --mobileDevice Pixel 2 --cwd /output/.tmpsdqb2yyl --statsFilename /output/crawl.json
from zimfarm.
Docker has multiple storage drivers.
I'm using ZFS because it works really well for Docker and hybrid disk+NVMe storage. Docker uses ZFS snapshots for layers. Compression+dedup is on for Docker layers and $ZIMFARM_ROOT/data
has compression. It's not an uncommon setup. df
and du
won't work like you think they would.
from zimfarm.
Related Issues (20)
- Again \u0000 cannot be converted to text
- Add capability to recreate a ZIM with updated metadata
- Add concept of scraper release channels HOT 1
- Simplify warehouse list HOT 4
- Zimfarm worker: add support for multiple outgoing IPs HOT 4
- Fix sotoki scraper configuration
- Watcher is not detecting updated files when SE is fixing issues HOT 3
- Seems impossible to specify a tag HOT 5
- Upload logs and artifacts even when task is cancelled
- Name resolution errors in StackExchange watcher
- Configuration of Zimfarm local instance HOT 1
- Installing monitoring of a task is failing HOT 2
- StackExchanger watcher regularly restarts after connection reset by peer errors HOT 1
- Is the worker incorrectly checking the space already consumed by the Zimfarm? HOT 8
- Collect zimfarm usage statistics
- Some Zimfarm task are never finishing while the process is obviously finished HOT 5
- Adapt Zimfarm to MWoffliner 1.14 HOT 3
- Ensure we report MiB/GiB/... values everywhere
- Migrate to MWoffliner 1.14 HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zimfarm.