dogsheep / hacker-news-to-sqlite

Create a SQLite database containing data pulled from Hacker News

Dogsheep

Dogsheep is a collection of tools for personal analytics using SQLite and Datasette.

Big internet companies know a lot about us. By exporting that data back out of them we can see what they know and maybe learn something interesting about ourselves.

Read more about Dogsheep on my blog: simonwillison.net/tags/dogsheep

Watch Personal Data Warehouses: Reclaiming Your Data for a demo of Dogsheep in action.

Dogsheep tools

These tools, maintained by the Dogsheep project, let you export your data into a SQLite database for further analysis.

• twitter-to-sqlite uses the Twitter API or Twitter's "export your data" zip archive to create a database of your tweets, followers, people you follow, favourited tweets and much more besides.
• healthkit-to-sqlite converts your Apple HealthKit export into a database, with details of your heart rate, workouts, step count and more.
• swarm-to-sqlite imports your checkin history from Foursquare Swarm.
• apple-notes-to-sqlite saves Apple Notes data to a database.
• inaturalist-to-sqlite imports the wildlife and plants you have spotted on iNaturalist.
• google-takeout-to-sqlite imports data from Google Takeout, which can include your Google Maps location history, your search history on Google and more to come.
• genome-to-sqlite imports your genome from 23andMe into a SQLite database.
• github-to-sqlite imports repositories you have created or starred on GitHub.
• pocket-to-sqlite imports articles you have saved using Pocket.
• hacker-news-to-sqlite imports stories and comments posted to Hacker News.
• dogsheep-photos imports metadata about photos from Apple Photos, including machine learning labels. See Using SQL to find my best photo of a pelican according to Apple Photos.
• dogsheep-beta is a search index which combines data from all of the other sources into a single faceted search interface.
• evernote-to-sqlite converts exported Evernote files in the ENEX format into a SQLite database.

Tools by other developers

These tools help bring the Dogsheep philosophy to life.

• beeminder-to-sqlite by Ben Congdon saves your goals and data points from Beeminder.
• goodreads-to-sqlite by Tobias Kunze imports your reading history from Goodreads.
• pinboard-to-sqlite by Jacob Kaplan-Moss saves your bookmarks from Pinboard.
• todoist-to-sqlite by Ben Congdon saves your tasks and projects from Todoist.
• parkrun-to-sqlite by Mark Woodbridge imports your parkruns.

hacker-news-to-sqlite's Issues

Command for fetching Hacker News threads from the search API

I want to be able to fetch every item for a domain, e.g. https://news.ycombinator.com/from?site=simonwillison.net

Set up full text search

Should run against title and text in items, and about and id in users.

Use HN algolia endpoint to retrieve trees

The trees command currently has to make a request for every single comment. Algolia have an endpoint that bundles the entire thread together into a single request.

https://hn.algolia.com/api/v1/items/ID

Here's an example that loads quickly, with about 50 comments: https://hn.algolia.com/api/v1/items/27941108

It doesn't appear to use pagination at all - if a thread is big then the response is big.

I ran this search to find some stories with more than 1000 comments: https://hn.algolia.com/api/v1/search?tags=story&numericFilters=num_comments%3E=1000

Here's one: https://news.ycombinator.com/item?id=25015967 with 4759 comments. Hitting the API takes 41s and returns 3.7 MB of JSON!

wget 'https://hn.algolia.com/api/v1/items/25015967'  0.03s user 0.04s system 0% cpu 41.368 total
/tmp % ls -lah 25015967 
-rw-r--r--  1 simon  wheel   3.7M Jul 24 20:31 25015967

Retrieve the top-level story for a comment

I think that each comment inserted into the database should include a column onstory that contains the ID of the story on which the comment was made. This is exactly equivalent to the link after "on:" at the top of an HN comment page (example). We could do this either by directly retrieving the HTML page and using Beautiful Soup to find that link, or alternatively recurse up the tree in the Firebase API using the parent field (probably using functools.lru_cache in case a person has commented a bunch of times on the same story).