isabella232 / sentry-github-actions-app

This app allows your organization to trace Github Actions with Sentry. You can use this to get insight of what parts of your CI is slow or failing often

Sentry Github Actions app

DISCLAIMER: This app/sdk are still in experimental mode. Please file an issue and ask of the current state before going ahead and using it.

This app allows your organization to trace Github Actions with Sentry. You can use this to get insights of what parts of your CI are slow or failing often. It works by listening to a Github workflow events via a webhook in your repository (You may be able to create a Github App of your own to avoid using personal or bot tokens). These events are then stored in Sentry as performance transactions.

Code explanation:

• github_sdk.py has the generic logic to submit Github jobs as transactions. Eventually this file could be released separatedly.
• github_app.py contains code to handle a Github App installation.
• web_app_handler.py: Web app logic goes here.
• main.py contains the code to respond to webhook events.

You can set up this app following the instructions under "Self-hosted". We also have a Github App to increase the security, however, it can only be installed in the getsentry Github org. In the future, we may make it pubclicly available. There's few things we need to figure out before we do.

Github App

NOTE: Currently, it is only used internally. We do not have a way of mapping CI events from one org to a specific Sentry DSN. See milestone to see required work.

Here are the list of values needed to set up the Github App and the associated env variables:

• App ID - GH_APP_ID
  • When you load the app in Github, you will see it listed in the page
• Installation ID - GH_APP_INSTALLATION_ID
  • Once you install the app under your Github org, you will see it listed as one of your organizations integrations
  • If you load the page, you will see the ID as part of the URL
• Private key - GH_APP_PRIVATE_KEY
  • When you load the Github App page, at the bottom of the page under "Private Keys" select "Generate a private key"
  • A .pem file will be downloaded locally.
  • Convert it into a single line value by using base64 (base64 -i path_to_pem_file)
  • In GCP, you can store this value more securely under Github Secrets (rather than an env variable)

For local development, you need to make the App's webhook point to your ngrok set up, create a new private key (a .pem file that gets automatically downloaded when generated).

NOTE: Do not forget to delete the private key when you are done.

Set up

When creating the Github App for the first time, these are non-default relevant changes made:

• Configure the Webhook URL to point to the GC deployment
• Set the Webhook Secret to be the same as the GC deployment
• Set the following permissions:
  • Actions: Workflows, workflow runs and artifacts -> Read-only
    • Specific events: Workflow job

After the creation of the app:

• Click on Generate a private key
  • Run base64 -i <path_to_download_file>
  • Store the value in Google Secrets
  • Delete the file just downloaded
• Add to the deployment the variable GH_APP_ID
• Select "Install App" and install the app on your org
  • Grant access to "All Repositories"
• Look at the URL of the installation
  • Add to the deployment the variable GH_APP_INSTALLATION_ID

Self-hosted

Sentry

In Sentry.io (or self-hosted install):

• Create a project to track errors of the app itself (APP_DSN in section below)
• Create a project to track Github jobs (SENTRY_GITHUB_DSN in section below)

Create a Github personal token

NOTE: This has the security risk that the token could be used to impersonate you.

• Create a personal token
  • You do not need to give it any scopes
  • Take note of the token as you will need to define it as GH_TOKEN in the next section
  • After creating the token, some orgs require authorizing tokens, thus, select "Configure SSO" if you see it
• If this is for development, you can place the token in a file named .env as GH_TOKEN

Deployment set up

• Deploy the app to a service (e.g. GCP) and make it publicly reachable
• Take note of the app's URL and use it when creating the Github webhook
• Environment variables:
  • APP_DSN: Report errors to Sentry of the app itself
  • GH_WEBHOOK_SECRET: Secret shared between Github's webhook and your app
  • GH_TOKEN: This is used to validate API calls are coming from Github webhook
  • SENTRY_GITHUB_DSN: Where to report Github job transactions
  • LOGGING_LEVEL (optional): To set the verbosity of Python's logging (defaults to INFO)

Optional: Once you have a deployment, you may wish to go back to Sentry and whitelist the deployment.

Github webhook

Create a Github webhook for a repo (or an org):

• Under settings select "Webhooks":
• Choose "Add webhook"
• Choose application/json
• Choose workflow events
• Add a secret with python3 -c 'import secrets; print(secrets.token_urlsafe(20))' on your command line
  • Set GH_WEBHOOK_SECRET as an env variable in your deployment
  • For more info, read Github docs
• Enter the URL of the deployed app
  • Use an ngrok URL for local development (read next section)
• Save the webhook

Note: Webhook events will immediately be sent. Disable the hook if you're not ready.

Local development

Prerequisites:

• ngrok (for local dev)

Set up:

python3 -m venv .venv
source .venv/bin/activate
pip install wheel
pip install -r requirements.txt -r requirements-dev.txt

You can ingest a single job without webhooks or starting the app by using the cli. For example:

# This is a normal URL of a job on Github
python3 cli.py https://github.com/getsentry/sentry/runs/5759197422?check_suite_focus=true
# From test fixture
python3 cli.py tests/fixtures/jobA/job.json

Steps to ingest events from a repository:

• Install ngrok, authenticate and start it up (ngrok http 5001)
  • Take note of the URL
• Create a Github webhook for the repo you want to analyze
  • Choose Workflow jobs events & make sure to choose application/json
  • Put Ngrok's URL there

Table of commands:

Command	Description
flask run -p 5001	Start the Flask app on http://localhost:5001
pre-commit install	Install pre-commit hooks
pytest	Run Python tests
pytest --cov=src --cov-report=html	Generate code coverage.

Sentry staff info

Google Cloud Build will automatically build a Docker image when the code merges on armenzg/github-actions-app:main. Logging to Google Cloud Run and deploy the latest image.