Watcher is not detecting updated files when SE is fixing issues about zimfarm

[benoit74]

Btw, this could be an elegant solution to fix the fact that all dumps currently stored on S3 are still from April 2 and "corrupted"

[rgaudin]

So it's clear for everybody, the watcher didn't pick up the fixed version because we identify a release as a YYYY-MM string given updates are expected every quarter.

This string is computed from the Last-Modified header of the Sites.xml file and compared to the one we stored in S3.
An update within the same month cannot be detected this way.

[benoit74]

Here the dumps have been updated but the Sites.xml has not been updated at-all, so current watcher behavior is not aligned (anymore) with SE practices.

I suggest to:

• stop checking the version of "Sites.xml" (we do not care about it indeed)
• check the Last-Modified header of every individual archive
• store the whole Last-Modified header in S3 metadata (instead of the simplified version)
• use this whole Last-Modified header for taking the decision to download the archive again