httparchive / hosts

The HTTP Archive tracks how the Web is built

!! Important: This repository is deprecated. Please see HTTPArchive/httparchive.org for the latest development !!

This repo contains the source code powering the HTTP Archive data collection.

What is the HTTP Archive?

Successful societies and institutions recognize the need to record their history - this provides a way to review the past, find explanations for current behavior, and spot emerging trends. In 1996 Brewster Kahle realized the cultural significance of the Internet and the need to record its history. As a result he founded the Internet Archive which collects and permanently stores the Web's digitized content.

In addition to the content of web pages, it's important to record how this digitized content is constructed and served. The HTTP Archive provides this record. It is a permanent repository of web performance information such as size of pages, failed requests, and technologies utilized. This performance information allows us to see trends in how the Web is built and provides a common data set from which to conduct web performance research.

hosts's Issues

Evaluate pshht

Alternative / something we could use?
https://github.com/dhs-ncats/pshtt

Alexa Top 1M is no longer. Replacement?

https://twitter.com/Alexa_Support/status/800755671784308736

Alternatives

Quantcast Top 1M US sites

Directly Measured: These properties allow Quantcast to monitor their traffic with Quantcast Measurement. Therefore, traffic numbers for these sites are highly accurate and not estimated. Learn more.

We collect directly measured data from the millions of web destinations and mobile apps controlled by Quantified publishers. All the data we collect is anonymous and contains no personally identifiable information (PII).

Majestic Million CSV

The Majestic Million is a list of the top 1 million website in the world, ordered by the number of referring subnets. A subnet is a bit complex – but to a layman it is basically anything within an IP range, ignoring the last three digits of the IP number.

ahrefs

Another crawl based service. I don't see any free ranking dumps though?

SimilarWeb Top sites

Top 50 for free.. requires paid account to see more.

---

Others we could or should consider?