simonw / paginate-json

Command-line tool for fetching JSON from paginated APIs

paginate-json

CLI tool for retrieving JSON from paginated APIs.

Currently works against APIs that use the HTTP Link header for pagination. The GitHub API is the most obvious example.

Usage: paginate-json [OPTIONS] URL

  Fetch paginated JSON from a URL

Options:
  --version                Show the version and exit.
  --nl                     Output newline-delimited JSON
  --jq TEXT                jq transformation to run on each page
  --accept TEXT            Accept header to send
  --sleep INTEGER          Seconds to delay between requests
  --silent                 Don't show progress on stderr
  --show-headers           Dump response headers out to stderr
  --header <TEXT TEXT>...  Send custom request headers
  --help                   Show this message and exit.

The --jq option only works if you install the optional pyjq dependency.

Works well in conjunction with sqlite-utils. For example, here's how to load all of the GitHub issues for a project into a local SQLite database.

paginate-json \
    "https://api.github.com/repos/simonw/datasette/issues?state=all&filter=all" \
    --nl | \
    sqlite-utils upsert /tmp/issues.db issues - --nl --pk=id

You can then use other features of sqlite-utils to enhance the resulting database. For example, to enable full-text search on the issue title and body columns:

sqlite-utils enable-fts /tmp/issues.db issues title body

You can use the --header option to send additional request headers. For example, if you have a GitHub OAuth token you can pass it like this:

paginate-json https://api.github.com/users/simonw/events \
  --header Authorization "bearer e94d9e404d86..."

paginate-json's Issues

--silent option for disabling output of information to stderr

It would turn off these lines:

paginate-json/paginate_json/cli.py

Lines 27 to 28 in 65bf74e

	for chunk in paginate(url, jq, accept):
	click.echo(len(chunk), err=True)

paginate-json/paginate_json/cli.py

Lines 39 to 40 in 65bf74e

	while url:
	click.echo(url, err=True)

Handle relative URLs in Link header

It would be super-handy if this tool were able to handle relative URLs in the link header; something like:

Link: </widgets?page=2>;rel="next"

which would be joined with the original URL. This would be relatively straightforward to implement using requests.compat.urljoin:

diff --git a/paginate_json/cli.py b/paginate_json/cli.py
index 4be0fb4..555577d 100644
--- a/paginate_json/cli.py
+++ b/paginate_json/cli.py
@@ -63,7 +63,11 @@ def paginate(
                 json.dumps(dict(response.headers), indent=4, default=repr), err=True
             )
         try:
-            url = response.links.get("next").get("url")
+            # Get the 'next' link
+            next = response.links.get("next").get("url")
+
+            # Resolve the potentially-relative URL to be an absolute URL
+            url = requests.compat.urljoin(url, next)
         except AttributeError:
             url = None
         if jq:

Thoughts? Would you be open to a PR that implements this?

--show-headers option for dumping HTTP headers to stderr

Useful for seeing things like rate-limiting headers.

--header multi-option for adding additional headers

In case you want to send e.g. an Authorization: Bearer xxx header.

--sleep option for sleep between requests