asadmansr / firebase-test-lab-action

A GitHub Action to test mobile applications using Firebase Test Lab.

Firebase Test Lab GitHub Action

A GitHub Action to test mobile applications (Android, iOS) using Firebase Test Lab.

Inspired by the 2020 GitHub Actions Hackathon.

Introduction

Testing mobile applications can be a challenge. With Firebase Test Lab, testing becomes much easier whether it's validating new changes on a continuous integration (CI) pipeline or tracking down bugs on specific devices. This GitHub Action automates the setup of the gcloud command line tool and provides an easy interface to start testing quickly.

Pre-requisites

1. Service Account: A service account is a special kind of account with specific permissions to authenticate with the Cloud Platform when used on a virtual machine for continuous integration.

2. ARG SPEC File: A YAML argument file that lists out all of the configurations for Firebase Test Lab. In this file, you can specify the test APK, filter the tests, select virtual or physical devices and indicate the type of test to perform.

Usage

workflows/main.yml:

name: Android CI
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      # Check out the repository
      - uses: actions/checkout@v2

      # Run the Firebase Test Lab Action
      - name: Run tests on Firebase Test Lab
        uses: asadmansr/[email protected]
        with:
          arg-spec: 'tests.yml:android-pixel-4'
        env:
          SERVICE_ACCOUNT: ${{ secrets.SERVICE_ACCOUNT }}

tests.yml:

android-pixel-4:
  type: instrumentation
  app: app-debug.apk
  test: app-debug-test.apk
  device:
    - model: flame
      version: 29
      locale: 'en'
      orientation: portrait

The following usage comes with additional instructions regarding the input and environment variables that can be found in the Simple Usage Documentation.

Currently, this GitHub Action only runs Android tests. Support for iOS coming soon.

Inputs

arg-spec

YAML file that contains configuration for Firebase Test Lab. Format must be ARG_FILE:ARG_GROUP_NAME. Required

SERVICE_ACCOUNT

Copy-paste the content of the JSON-formatted service account file in GitHub's secret variables in settings. Required

Contributing

Are you facing an issue? Have some questions? Would like to implement a new feature? Learn more about our contributing guidelines.

Licence

The project is released under the MIT License.

firebase-test-lab-action's Issues

Action output: link to a run

Your action works great. I'm integrating it with Slack notification if the test on Test Lab fails. It would be useful to have a output variable with the test run, so I can link it directly from the Slack message.

Set option for Enable Orchestrator

i want to run my Espresso Test with Orchestrator Enable, google have given below argument to set, so how to set the option?
--use-orchestrator

How and where do we add the target file to be tested

Hey,

I would like to know how and where would i mention the path to the target .dart file where i have my tests?

Example for ios project?

It's working very well for android. Would it work too for ios projects?

The firebase command is not that different:
gcloud firebase test ios run --test PATH/TO/MyTests.zip \ --device model=iphonexr,version=13.2,locale=en_US,orientation=portrait --xcode-version=12.1

Would this work with the current action, something like?

tests.yml

ios-iphonexr:
  type: instrumentation
  app: ''
  test: iosOutputZip.zip
  device:
    - model: iphonexr
      version: 13.2
      locale: 'en_US'
      orientation: portrait```

Doesn't fail the build when the tests fail

Describe the bug
Doesn't fail the build when the tests fail

Expected behavior
The github action workflow halting since the tests failed.
Instead of that the workflow keeps going and succeeds, even-though the test failed.

Is that expected behaviour? How can I make it so that the workflow fails?