stats download for November is empty about legacy.httparchive.org

[rviscomi]

Looks like stats is 217KB as of the 22nd:

217K Nov 22 20:23 httparchive_stats.gz

However httparchive_schema.sql and httparchive_Nov_1_2019_requests.csv.gz are both 0 bytes so something is amiss.

[Themanwithoutaplan]

Yes, at some point the file did turn up, but as you note, there were other problems. I hope I'm not the only person who finds these files useful.

[rviscomi]

I looked into the issue on Wednesday and unfortunately I wasn't able to get it to work. Something in the desktop requests CSV pipeline is causing a full disk issue. And I think at the end of the job it nuked the requests table completely, so we'd need to repopulate it from a backup. I'm also getting issues in the Dataflow pipeline with the desktop HAR results... Unfortunately, I'm currently OOO until January so I don't have time to look into these issues. Maybe @paulcalvano @pmeenan or @igrigorik could have a look.

TBH you're the only person I've heard from you uses the raw files and while they are part of the deprecated legacy code base, the new pipeline does depend on them too. @Themanwithoutaplan the mysql dump (not CSV) should still be available if you're able to make use of that.

[Themanwithoutaplan]

I actually only need the pages files but I do need these as CSV because I put them into Postgres for analysis using a modified, more efficient schema. FWIW pages vs. requests is a good example of SQL vs. NoSQL. Analysis of requests is very ad hoc and well-suited to map/reduce whereas the pages data is useful for more classic projections.

[Themanwithoutaplan]

Looks like this has now been resolved. Thanks.