mozilla / tracking-protection-issues-exporter

Helper script to export tracking protection GitHub issues created by https://github.com/mozilla/tracking-protection-issues

Tracking Protection Issues Exporter

NodeJS script to export GitHub issues generated from Firefox's ETP breakage reporting UI. Issues are created by the tracking-protection-issues middleware. This script fetches issues (+comments) via the GitHub API, parses them into individual reports and stores them in a MongoDB database.

Due to GitHub's API limits downloading a large amount of issues and comments can take a long time. The script will gracefully handle API failures. Failed requests will be retried. When hitting the rate limit the script will wait for the next (hourly) reset before continuing.

Setup

Tested with NodeJS v12

Install dependencies via npm:

npm install

The script needs access to a MongoDB server to use as the storage backend. You can use Docker Compose with the docker-compose.yml config to start a local development server:

docker-compose up mongo

Usage

cli.js implements a command line interface to download and convert issues:

node src/cli.js [command]

Commands:
  fetch    Fetch ETP issues from GitHub and import them into
                               a database.
  convert  Convert ETP GitHub issues into reports.

Options:
  --help                     Show help                                 [boolean]
  --version                  Show version number                       [boolean]
  --dbHost                   MongoDB database host
                                           [string] [default: "localhost:27017"]
  --dbUser                   MongoDB username                [string] [required]
  --dbPassword               MongoDB password                [string] [required]
  --dbDatabaseName           MongoDB database name
                                         [string] [default: "etp-issues-export"]
  --dbCollectionNameIssues   Name of the db collection to store issues in.
                                                    [string] [default: "issues"]
  --dbCollectionNameReports  Name of the db collection to store reports in.
                                                   [string] [default: "reports"]

Pass --help to the sub-commands to see all options. For example node src/cli.js fetch --help.

Pass options via environment

Options can also be defined via environment variables, or an .env file in the repositories root directory:

EXPORT_GH_AUTH_TOKEN=<token>
EXPORT_GH_REPO_USER=<user>
EXPORT_GH_REPO_NAME=<repoName>
EXPORT_GH_API_MAX_RETRY=50
EXPORT_DB_HOST=localhost:27017
EXPORT_DB_USER=root
EXPORT_DB_PASSWORD=123456
EXPORT_DB_DATABASE_NAME=etp-reports
EXPORT_DB_COLLECTION_NAME_ISSUES=issues
EXPORT_DB_COLLECTION_NAME_REPORTS=reports

tracking-protection-issues-exporter's Issues

Script crashes instead of pausing when hitting the API limit

Once hitting the limit during import the script throws. Seems like the rate limit code is not working correctly anymore.

RequestError [HttpError]: API rate limit exceeded for user ID [ID]. If you reach out to GitHub Support for help, please include the request ID [ID].
    at /home/parallels/Downloads/tracking-protection-issues-exporter/node_modules/@octokit/request/dist-node/index.js:86:21
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Job.doExecute (/home/parallels/Downloads/tracking-protection-issues-exporter/node_modules/bottleneck/light.js:405:18) {
  status: 403,
  response: {
    url: 'https://api.github.com/repos/[user]/[repo]/issues/[id]/comments?per_page=100&sort=created&direction=desc',
    status: 403,
    headers: {
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
      connection: 'close',
      'content-encoding': 'gzip',
      'content-security-policy': "default-src 'none'",
      'content-type': 'application/json; charset=utf-8',
      date: 'Wed, 08 Nov 2023 16:17:54 GMT',
      'github-authentication-token-expiration': '[DATE] UTC',
      'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
      server: 'GitHub.com',
      'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
      'transfer-encoding': 'chunked',
      vary: 'Accept-Encoding, Accept, X-Requested-With',
      'x-accepted-oauth-scopes': 'repo',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'deny',
      'x-github-media-type': 'github.v3; format=json',
      'x-github-request-id': '[ID]',
      'x-oauth-scopes': 'repo',
      'x-ratelimit-limit': '5000',
      'x-ratelimit-remaining': '0',
      'x-ratelimit-reset': '1699462175',
      'x-ratelimit-resource': 'core',
      'x-ratelimit-used': '5000',
      'x-xss-protection': '0'
    },
    data: {
      message: 'API rate limit exceeded for user ID [ID]. If you reach out to GitHub Support for help, please include the request ID [ID].',
      documentation_url: 'https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting'
    }
  },
  request: {
    method: 'GET',
    url: 'https://api.github.com/repos/[user]/[repo]/issues/[id]/comments?per_page=100&sort=created&direction=desc',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': 'octokit-rest.js/1.7.0 octokit-core.js/3.5.1 Node.js/20.9.0 (linux; arm64)',
      authorization: 'token [REDACTED]'
    },
    request: { hook: [Function: bound bound register] }
  }
}

Time range issue fetching

This is to support updating an existing dataset without having to download all issues and comments again.

There is a since API param we could use: https://octokit.github.io/rest.js/v18#issues-list-for-repo
It would have to be applied to both the issues and the comments query.

Make issue import (fetch) resumable

Fetching issue comments is already resumable, but fetching issues isn't. MongoDB will throw for duplicate insertions.