rviscomi / har.fyi Goto Github PK

Most of the Web Almanac queries reference the almanac dataset, which is the predecessor to the all.pages and all.requests tables. The schemas are very similar, but some massaging is still needed to convert an Almanac query to use the newer tables.

This conversion is necessary if someone wanted to rerun a query against a newer dataset. The almanac tables were manual snapshots for a particular crawl in the year, and all chapters were aligned to that crawl. Because the tables were manually created, there are no corresponding tables for the crawls after 2022.

We should write a new guide for this exact use case: converting almanac queries to use either all.pages or all.requests. Also, most of the CSS chapter queries reference httparchive.almanac.parsed_css which lives on as the httparchive.experimental_parsed_css dataset, but it's worth noting that due to a processing error, the tables only go back to September 2023 (hence "experimental").

My longshot goal for this work is to enable ML to consume this guide for training and take any Web Almanac query as input and spit out a perfectly functional query for any of the modern crawls. We'll see if our documentation (and the AI technology) is up to the challenge!