every index.html code brocken since today about wayback-machine-downloader

[gingerbeardman]

@ckought I do not see this, so I wonder what version of wayback_machine_downloader are you using and has it been modified in any way?

Use the one in #280 or see this comment #265 (comment)

[ckought]

@gingerbeardman That fixed the issue. I've had it running for about 48 hours, with 25K downloads with no corrupted downloads.

The version of wayback_machine_downloader.rb I had been using was the version from the automatic install using "gem install wayback_machine_downloader".

[gingerbeardman]

My index files are downloaded and look as I expect them to.

Feel free to post a sample URL where you are seeing this and I'll check.

[ckought]

I'm getting this for about a third of the pages that I'm grabbing. No rhyme or reason for what ones are corrupted. It's happening to txt and html files, and probably other file types too (I know some jpg files are getting corrupted, but I'm not sure on css or js or others).

It looks to me like whatever code is being used to strip off the archive.org code at the top is causing it. It's like the code is downloading the page, stripping off the archive.org code, something goes wrong, and then it writes the garbage file to disk thinking the file is just fine.

[ckought]

Update:

Been doing some digging, and it looks like every corrupted file, no matter if it's html, txt, jpeg, css, all start with one of these three sets of HEX characters:

1f 8b 08 00 00 00 00 00 00 03
1f 8b 08 00 3f 3f 3f 00
1f 8b 08 00 4f 3f 3f 00

Still not sure what's actually causing the files to corrupt though. It does seem to get worse the longer larger the website is and the therefore the longer the script is running, so that may have something to do with it.