mozilla / burnham

Automated end-to-end testing for Mozilla's Glean telemetry. 👩‍🚀

burnham 👩‍🚀📈🤖

❌ As of August 4, 2022, the burnham end-to-end test suite was disabled on Airflow. The source code in this repository is no longer actively maintained and the repository was marked as read-only as a result. ❌

The burnham project is an end-to-end test suite that aims to automatically verify that Glean-based products correctly measure, collect, and submit non-personal information to the GCP-based Data Platform and that the received telemetry data is then correctly processed, stored to the respective tables and made available in BigQuery.

Blog post about the proof of concept: https://raphael.codes/blog/automated-end-to-end-tests-for-glean/

Overview

Glean SDK

For Mozilla, getting reliable data from our products is critical to inform our decision making. Glean is a product analytics & telemetry solution that provides a consistent experience and behavior across all of our products.

Find out more in our Data Documentation. 📝

burnham

We have developed a command-line application based on the Glean SDK Python bindings for producing test data as part of the automated end-to-end test suite. The burnham application submits Glean pings to the Data Platform which validates and stores these pings in BigQuery tables.

You can find the code for the burnham application in the application directory. 👩‍🚀

burnham-bigquery

We also developed a test suite based on the pytest framework that dynamically generates tests. Each test runs a specific query on BigQuery to verify a certain test scenario. After the test session finished, we then store the results in a designated BigQuery table with ID burnham_derived.test_results_v1.

The test suite code is located in the bigquery directory. 📊

telemetry-airflow

We build and push Docker images for both burnham and burnham-bigquery on CI for pushes to the main branch of this repository. The end-to-end test suite is configured as a DAG on telemetry-airflow on the Data Platform and scheduled to run daily. It runs several instances of a burnham Docker container to produce Glean telemetry, uses Airflow sensors to wait for the data to be available in the various burnham live tables, and then runs burnham-bigquery to verify the results.

Please see the burnham DAG for more information. 📋

Redash

We created two scheduled queries that read the results from the burnham_derived.test_results_v1 table from the past 4 days. The queries run daily at 02:00 UTC after telemetry-airflow ran the burnham DAG.

Mozilla employees can find this information in the burnham dashboard.

Development status

We successfully completed the proof of concept and are now running burnham and burnham-bigquery in production. We are now actively working on developing additional test scenarios. 🚀

Requirements

Both burnham and burnham-bigquery run on Python 3.7 on Debian.

Integration points

The burnham project has integration points with the Glean SDK team, the Data Platform team and the Ecosystem Test Engineering team.

Community Participation Guidelines

This repository is governed by Mozilla's code of conduct and etiquette. Please read the burnham Code of Conduct.

Project theme

The project theme is inspired by Michael Burnham, the fictional protagonist on the web television series Star Trek: Discovery portrayed by Sonequa Martin-Green. Burnham is a science specialist on the Discovery. She and her crew do research on spore drive technology and complete missions in outer space and these themes of scientific exploration and space travel are a perfect fit for this project. 👩‍🚀

burnham's Issues

Create Docker image with CLI app

We will be running the end-to-end tests as Docker containers in Airflow. For that we need to create a Docker image with the CLI app that submits telemetry to a server specified by the environment.

Test ping names

Is your feature request related to a problem? Please describe.
We recently saw a bug where custom pings that were named in kebab-case where being dropped by the schema at ingestion because they were invalid against the schema. We plan to fix the schema, but it would be nice to have a test to prevent this kind of thing in the future.

Describe the solution you'd like
Send some custom pings with various kinds of names and make sure they are ingested correctly.

Explicitly, we should support:

• kebab-case
• snake_case
• numbers001

Explicitly not supported are capital letters or names over 30 characters.

Additional context

mozilla-services/mozilla-pipeline-schemas#470
mozilla/glean#616

Templated run ID is not rendered in burnham-bigquery SQL

Update CLI flag for telemetry to be enabled by default

We use the CLI application to test telemetry, so it makes sense to enable that by default.

Set up actions workflow to publish to Docker registry

https://github.com/features/packages 📝

Develop test framework for verifying data

The framework needs to support specifying the expected values for live and error tables.

Update DAG to changed error reporting behavior in Glean 🚧

Glean v31.5.0 changed the error reporting behavior. String values that are too long now record invalid_overflow rather than invalid_value. We need to update the DAG to check for the invalid_overflow instead.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1664119

Should Burnham use the client activity API?

Is your feature request related to a problem? Please describe.

Glean now provides the client activity API.
It is not yet implemented for Python.
This allows embedding applications to trigger behavior based on client activity.
On mobile this maps to foregrounding (active) and backgrounding (inactive).
For Desktop this maps to certain activity within a certain amount of time.

The API triggers the baseline and events ping, both otherwise internal pings.

Describe the solution you'd like

Would it be useful for burnham to use this API to simulate activity and thus test Glean's expected behavior around those pings?

Develop fake data-platform endpoint

We need a minimal HTTP service for developing the end-to-end tests before we submit to the actual data-platform.

Unpin Python bindings for Glean SDK 📦

See https://github.com/mozilla/burnham/blob/main/application/setup.py#L36

Develop additional unit tests for burnham 🚀

We currently run flake8 and mypy on the burnham code as well as one smoke test for the CLI. I suggest we develop additional unit tests to increase the code coverage and reduce the risk of breaking burnham. I also suggest we add an additional tox environment to report the code coverage and fail the tox environment if the code coverage falls under a certain value. This will make it easier for newcomers to contribute to the project.

See also #93

Update burnham-bigquery to support run identifier as CLI option

The test run UUID is templated in Airflow. Remove it from the JSON-encoded test run information and add a an extra CLI option for that to ensure the template is rendered properly.

Testing new and upcoming Glean SDK releases

We recently discussed in the cross-team meeting whether we could run the existing tests also with older version of the Glean SDK used by products as well as the latest main branch.

This issue is for exploring this idea further and scoping out the work required to achieve this.

Update submission_timestamp in burnham DAG 📈

The burnham DAG currently uses:

WHERE
  submission_timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 3 HOUR)
  AND metrics.uuid.test_run = @burnham_test_run

When we restart the verify_data task outside of this time window task, we don't get any results back.

@fbertsch suggested to use a parameter for submission_timestamp.

Document how to use a custom Glean SDK wheel 📝

When working on bug fixes for the Glean Python SDK, we would like to replace the Glean SDK in the burnham Docker image with a custom wheel, built locally.

Dependency issue with glean-parser 📦

The Docker build step on CircleCI fails after upgrading glean-parser in #84

RUN python -m pip install --no-index --find-links=/tmp/wheels/ burnham

ERROR: Could not find a version that satisfies the requirement glean-parser==1.28.6 (from glean-sdk>=32.3.1->burnham) (from versions: 1.29.0)
ERROR: No matching distribution found for glean-parser==1.28.6 (from glean-sdk>=32.3.1->burnham)

Move Python CLI application to subdirectory

This will help us separate between project files from Python package distribution files.

Transfer project to Mozilla GitHub org

🚛

Write documentation for test framework

• High level overview
• Integration points
• Requirements

Disable GitHub actions

GitHub actions are currently disabled for repos in the organization.

Create docker-compose for developing burnham

See #2 and #3.

Update email alert receivers 📩

See https://github.com/mozilla/telemetry-airflow/blob/master/dags/burnham.py#L18

Docker build for burnham broken 🚧

We haven't made any changes to the Docker build configuration for burnham since the most recent successful build job. Here's what I found so far by comparing the CI logs:

Most recent successful build

pip 20.1.1
wheel 0.34.2
setuptools 47.3.1

Failing build

pip 20.1.1
wheel 0.34.2
setuptools 48.0.0

cc @jezdez

Pin to new Glean SDK release 📦

Wait for a new Glean SDK version to be released that includes the bug fix for core metrics.

cc @badboy

Request data review for metrics

See https://mozilla.github.io/glean/book/user/adding-new-metrics.html 🤖

Publish burnham to PyPI

We currently don't publish new releases of the burnham CLI to PyPI.

Do you think it would be useful to start doing this? 📦

Set up CI for CLI code checks

We deleted our GitHub Actions workflow because the feature is disabled for the org (see #19).

Update name for default Git branch

I suggest we rename master to a more inclusive name, as for example main.

Create new DAG in Airflow for tests

The DAG needs to run the burnham Docker image to produce Glean Telemetry, wait for Glean Telemetry to be available in Big Query tables, and run the Big Query CLI Docker image to verify the Glean Telemetry.

See https://github.com/mozilla/telemetry-airflow

Integrate Glean SDK into burnham CLI app

Write application that uses the Glean SDK to record and submit telemetry to the specified server.

Set up CD for CLI Docker image

We need to continuously build new Docker images for the CLI application and push to a Docker registry that we can pull from in Airflow.

For reference, see this Circle CI config created by @jklukas:
https://github.com/mozilla/forecasting/blob/master/.circleci/config.yml

Test for disabling upload and deletion-request ping

We started discussing a new test for the deletion-request ping in #55. Testing that pings are deleted from BigQuery tables following a deletion-request is complicated. However we can test that burnham doesn't submit pings after disabling the upload in the Glean SDK, that the client gets a new client ID assigned after re-enabling the upload in the Glean SDK and the BigQuery live tables contain the expected data.

Proposed test scenario

Produce data 👩‍🚀

(requires changes to burnham in this repository)

• burnham client records to space_travel:warp_drive metric
• burnham client submits discovery ping
• burnham client calls Glean.set_upload_enabled(False)
• burnham client records to space_travel:spore_drive metric
• burnham client submits discovery ping
• burnham client calls Glean.set_upload_enabled(True)
• burnham client records to space_travel:spore_drive metric
• burnham client submits discovery ping

Verify data 📈

(requires changes to the burnham DAG in telemetry-airflow)

• table discovery_v1 contains ping with space_travel:warp_drive metric values from client with ID A
• table discovery_v1 does not contain ping with space_travel:spore_drive metric values from client with ID A
• table discovery_v1 contains ping with space_travel:spore_drive metric values from client with ID B
• table deletion_request_v1 contains ping client with ID A

Thoughts? Comments? Suggestions?

Update GitHub labels 📝

I suggest we update our GitHub labels to increase clarity around what an issue/pull-request is about.

(https://github.com/hackebrot/labels makes bulk editing labels easier)

Create labels for the different integration points:

• glean-sdk
• telemetry-airflow

Create labels to flag the involved burnham components:

• application
• bigquery
• fake-data-platform

Split the feature label into:

• capability (for example: improved logging)
• test scenario (for example: a new test for the deletion-request ping)

Create labels for GitHub integrations:

• Dependabot
• Circle CI
• GitHub

Anything else? 🤔

Update code checks for burnham and burnham-bigquery 🤖

We recently merged a pull-request from Dependabot that introduced incompatible dependencies for burnham (#84). As a consequence we cannot push new Docker images to GKE because the Docker workflow fails.

@jklukas correctly pointed out in #88 (comment) that we need checks that exercise the build workflow.

We currently run the following code checks on CircleCI (see config file):

• flake8
• mypy
• pytest

All of these run in a tox environment. As a first step, I suggest we update the installation step to also install from requirements.txt so that any dependency conflict is detected there. I also think we should build the Docker images for pull-requests.

According to the documentation of the CircleCI orb that we use for the Docker workflows, we should be able to add a gcp-gcr/build-image step. Ideally it will only run if the respective subdirectories are modified.

Enable burnham DEBUG logging on Airflow 📝

The Airflow logs for burnham clients don't seem to contain DEBUG log messages from the burnham CLI and the Glean SDK. Enabling this will be helpful when diagnosing issues with the burnham DAG.

See https://github.com/mozilla/telemetry-airflow/blob/master/dags/burnham.py

Update project documentation 📝

There are a few things we could improve about the documentation.

Project README:

• Update Development Status (the project is still under active development, but completed the poc stage)
• Add section to overview with burnham_derived.test_results_v1 table
• Add link to Redash dashboard with test results
• Update telemetry-airflow section (we now query multiple tables and use multiple sensors)

Application README:

• Update Development Status (the application API is stable now)
• Note that we currently don't upload new burnham releases to PyPI

BigQuery README:

• Update Development Status (the project is still under active development, but completed the poc stage)

Decompress ping payloads in fake-data-platform

The Glean Python SDK is now submitting gzip compressed pings. 📦

See https://mozilla.github.io/glean/book/appendix/changelog.html?highlight=compression#v3100-2020-05-28

Set up GitHub code owners 💻

I'm wondering if it would be useful to set up GitHub owners to auto-request review from specific folks when certain files are modified. See https://github.blog/2017-07-06-introducing-code-owners/

Update Dockerfile to use args instead of environment variables

We use arguments for the GKEPodOperator in telemetry-airflow.

Test that pings are deleted following deletion-request

Verify that the Data Platform deletes Glean telemetry for a client.

This test is slightly different from other tests as we may need to run a sequence of steps:

• produce Glean telemetry
• verify that it's available in BigQuery tables
• submit deletion-request ping
• verify that the telemetry for this client has been deleted

Create Docker image for Big Query CLI

We need a Docker image that runs queries against the produced data in Big Query and match the data against expected values.

Test for AET ping 📊

Issue for scoping out the work for testing the AET ping.

Switch to Calendar Versioning

Knowing when the used burnham version was released will help us diagnose failing end-to-end tests. I suggest we follow the versioning scheme YY.MINOR.MICRO. 📆

Create dashboard for test runs 📈

This depends on the test run results and reports being available on BigQuery (#39).

Store test run results along with report 📝

We need to store the test report and test run information from the burnham-bigquery Docker image somewhere.

Let's discuss a good approach for that.

Duplicate documents in latest test run

The most recent burnham test run failed and I can see two rows with the same document_id in the burnham live tables resulting in incorrect labeled_counter metric values. 🤔

Update burnham-bigquery to expect run info as base64

We encountered issues passing the JSON-encoded test run information to the GKEPodOperator due to string escaping issues. Update the test framework to expect run info as base64.

Glean Python bindings don't support UUID metrics

We currently cannot use the UUID metric type for the test run ID.

Status checks are not displayed on pull-request UI 🚧

For some reason the status checks from our CircleCI pipeline are not displayed on pull-requests anymore.

See #97 for additional context.

Enable auto pull-requests for new Glean SDK releases 📦

We unpinned the Glean SDK version in #67. The next step is to automatically build and push an updated burnham Docker image to GKE for new releases of the Python bindings on PyPI. We have configured CircleCI to build and push Docker images for pushes to the main branch. Can we add a requirements.txt file and set up Dependabot to automatically open pull-requests against that file for new releases of the Glean SDK? 🤖

https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/