Deal with storage usage limits about zimfarm

[automactic]

#44 is trying to solve this issue. You can have queues like small, medium and large. And then a worker can say only tune in to small and medium queues.

However, I do not think it is possible to do it based on file size. Also how would you know about file size before a zim file is generated?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

@rgaudin This was the first requirement about dealing with different kind of requirements for the scrapers. Multiple queue was a first answer but we now deal with other problems (priorities). To my opinion my first proposal, if implementable, would have avoid the kind of problem we have now.

[rgaudin]

OK, I understand now how we got there. It reinforces that run-time server+client sides knowledge requirement for task assignment that I described in #202.

While we can add a way to provide that task-required size in the DB, we should know how we're gonna get that information. It can (and should as most wikis grow) be overestimated but if we're going to compare two numbers to decide whether a task can be ran, we need them both.

How do you see us estimating those sizes?

[kelson42]

How do you see us estimating those sizes?

It would be an heuristic, manual or automated.

[rgaudin]

Sure but doesn't help much. We're gonna have to start somewhere with a basic algorithm and adjust over time while allowing manual input for those we have more info on and where it matters (largest ones).

• we have ~1,200 schedules
• all of them have a small, medium or large info (very approximate, will have to be refined over time)
• we don't inspect disk usage and so don't report it during our task life cycle. should we?
• we don't have direct access to number of articles but could get that info, if of value. Is it?
• how does format and formats combination influence the disk usage?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.