GithubHelp home page GithubHelp logo

pkjmesra / flank Goto Github PK

View Code? Open in Web Editor NEW

This project forked from flank/flank

0.0 1.0 0.0 23.3 MB

Massively parallel Android and iOS test runner for Firebase Test Lab

License: MIT License

Shell 0.33% Kotlin 32.71% Java 34.48% JavaScript 0.60% HTML 31.26% CSS 0.06% TypeScript 0.54% Go 0.02%

flank's Introduction

Flank Build Status codecov

Flank is a massively parallel Android and iOS test runner for Firebase Test Lab.

Contributing

  • Install Oracle JDK 8
    • JDK 9 or later will not work
  • Use JetBrains Toolbox to install IntelliJ IDEA Community
  • Clone the repo git clone --recursive https://github.com/TestArmada/flank.git
    • git submodule update --init --recursive updates the submodules
  • Open test_runner/build.gradle.kts with IntelliJ IDEA Community

Features

Available now Late 2018 Early 2019
Test sharding Smart Flank Client/Server refactor
Cost reporting
Stability testing
HTML report
JUnit XML report

Exit Codes

Exit code Description
0 All tests passed
1 At least one test failed or inconclusive and all matrices finished.
2 At least one matrix not finished, usually a FTL error.

iOS example

Run test_runner/flank.ios.yml with flank to verify iOS execution is working.

  • cd test_runner/
  • ./gradlew clean assemble fatJar
  • java -jar ./build/libs/flank-*.jar firebase test ios run
# gcloud args match the official gcloud cli
# https://cloud.google.com/sdk/gcloud/reference/alpha/firebase/test/ios/run
gcloud:
  # results-bucket: tmp_flank
  record-video: true
  timeout: 30m
  async: false
  # project: delta-essence-114723
  # results-history-name: ios-history

  # test and xctestrun-file are the only required args
  test: ./src/test/kotlin/ftl/fixtures/tmp/EarlGreyExample.zip
  xctestrun-file: ./src/test/kotlin/ftl/fixtures/tmp/EarlGreyExampleSwiftTests_iphoneos12.1-arm64e.xctestrun
  xcode-version: 9.2
  device:
    - model: iphone8
      version: 11.2
      locale: en
      orientation: portrait

flank:
  # test shards - the amount of groups to split the test suite into
  # set to -1 to use one shard per test.
  testShards: 1
  # repeat tests - the amount of times to run the tests.
  # 1 runs the tests once. 10 runs all the tests 10x
  repeatTests: 1
  # always run - these tests are inserted at the beginning of every shard
  # useful if you need to grant permissions or login before other tests run
  test-targets-always-run:
    - a/testGrantPermissions
  # test targets - a list of tests to run. omit to run all tests.
  test-targets:
    - b/testBasicSelection

Android example

Run test_runner/flank.yml with flank to verify Android execution is working.

  • cd test_runner/
  • ./gradlew clean assemble fatJar
  • java -jar ./build/libs/flank-*.jar firebase test android run
# gcloud args match the official gcloud cli
# https://cloud.google.com/sdk/gcloud/reference/firebase/test/android/run
gcloud:
  results-bucket: tmp_flank
  record-video: true
  timeout: 30m
  async: false
  project: delta-essence-114723
  results-history-name: android-history

  # test and app are the only required args
  app: ../test_app/apks/app-debug.apk
  test: ../test_app/apks/app-debug-androidTest.apk
  auto-google-login: true
  use-orchestrator: true
  environment-variables:
    clearPackageData: true
  directories-to-pull:
    - /sdcard/screenshots
  performance-metrics: true
  test-targets:
    # supported are class, notClass, size, annotation, notAnnotation, package, notPackage, testFile, notTestFile
    # as described in https://developer.android.com/reference/android/support/test/runner/AndroidJUnitRunner
    - class com.example.app.ExampleUiTest#testPasses
  device:
    - model: NexusLowRes
      version: 28

flank:
  # test shards - the amount of groups to split the test suite into
  # set to -1 to use one shard per test.
  testShards: 1
  # repeat tests - the amount of times to run the tests.
  # 1 runs the tests once. 10 runs all the tests 10x
  repeatTests: 1
  # always run - these tests are inserted at the beginning of every shard
  # useful if you need to grant permissions or login before other tests run
  test-targets-always-run:
    - class com.example.app.ExampleUiTest#testPasses

CI integration

Download Flank from GitHub releases.

Stable

wget --quiet https://github.com/TestArmada/flank/releases/download/v3.0.0/flank.jar -O ./flank.jar
java -jar ./flank.jar android run

Snapshot (published after every commit)

wget --quiet https://github.com/TestArmada/flank/releases/download/flank_snapshot/flank.jar -O ./flank.jar
java -jar ./flank.jar android run

In CI, it may be useful to generate the file via a shell script:

cat << 'EOF' > ./flank.yml
gcloud:
  app: ../../test_app/apks/app-debug.apk
  test: ../../test_app/apks/app-debug-androidTest.apk
EOF

Gradle Plugin

Fladle is a Gradle plugin for Flank that provides DSL configuration and task based execution.

Authenticate with a service account

Follow the test lab docs to create a service account.

  • Save the credential to $HOME/.config/gcloud/application_default_credentials.json or set GOOGLE_APPLICATION_CREDENTIALS when using a custom path.
  • Set the project id in flank.yml or set the GOOGLE_CLOUD_PROJECT environment variable.

For continuous integration, base64 encode the credential as GCLOUD_KEY. Then write the file using a shell script. Note that gcloud CLI does not need to be installed. Flank works without any dependency on gcloud CLI.

Encode JSON locally.

base64 -i "$HOME/.config/gcloud/application_default_credentials.json" | pbcopy

Then in CI decode the JSON.

GCLOUD_DIR="$HOME/.config/gcloud/"
mkdir -p "$GCLOUD_DIR"
echo "$GCLOUD_KEY" | base64 --decode > "$GCLOUD_DIR/application_default_credentials.json"

Running with gcloud directly

flank.yml is compatible with the gcloud CLI.

  • gcloud firebase test android run flank.yml:gcloud
  • gcloud alpha firebase test ios run flank.ios.yml:gcloud

NOTE: You will need to activate gcloud's service account for the above commands to work.

Doctor

Use the doctor command to check for errors in the YAML.

  • flank firebase test android doctor
  • flank firebase test ios doctor

Check version

Flank supports printing the current version.

$ flank -v
v3.0-SNAPSHOT

FAQ

Access Not Configured. Cloud Tool Results API has not been used in project 764086051850 before or it is disabled.

This error means authentication hasn't been setup properly. See Authenticate with a service account in this readme.

How do I use Flank without typing long commands?

Add Flank's bash helper folder to your $PATH environment variable. This will allow you to call the shell scripts in that helper folder from anywhere.

With the flank shell script, you can use flank instead of java -jar flank.jar. Examples:

  • flank android run
  • flank ios run

With the update_flank.sh shell script, you can rebuild flank.jar.

Symbol is declared in module 'java.xml' which does not export package 'com.sun.org.apache.xerces.internal.dom'

Make sure you're using JDK 8 to compile Flank.

Test run failed to complete. Expected 786 tests, received 660

Try setting use-orchestrator: false. Parameterized tests are not compatible with orchestrator. Flank uses orchestrator by default on Android.

flank's People

Contributors

bootstraponline avatar renas avatar abhagupta avatar vanwagonet avatar macarse avatar doodla avatar runningcode avatar stephenbrough avatar bradylarson avatar rolyatwilson avatar vtsoup avatar mauin avatar hjain60 avatar artem-zinnatullin avatar sravanmedarapu avatar matthewrice345 avatar trevorneedham avatar geekdave avatar scompt avatar winterdroid avatar yogurtearl avatar miguelslemos avatar mrooney avatar rashiq avatar thalescm avatar azure-pipelines[bot] avatar csessa avatar zlippard avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.