Flank is a massively parallel Android and iOS test runner for Firebase Test Lab.
- Install Oracle JDK 8
- JDK 9 or later will not work
- Use JetBrains Toolbox to install
IntelliJ IDEA Community
- Clone the repo
git clone --recursive https://github.com/TestArmada/flank.git
git submodule update --init --recursive
updates the submodules
- Open
test_runner/build.gradle.kts
withIntelliJ IDEA Community
Available now | Late 2018 | Early 2019 |
---|---|---|
Test sharding | Smart Flank | Client/Server refactor |
Cost reporting | ||
Stability testing | ||
HTML report | ||
JUnit XML report |
Exit code | Description |
---|---|
0 | All tests passed |
1 | At least one test failed or inconclusive and all matrices finished. |
2 | At least one matrix not finished, usually a FTL error. |
Run test_runner/flank.ios.yml
with flank to verify iOS execution is working.
cd test_runner/
./gradlew clean assemble fatJar
java -jar ./build/libs/flank-*.jar firebase test ios run
# gcloud args match the official gcloud cli
# https://cloud.google.com/sdk/gcloud/reference/alpha/firebase/test/ios/run
gcloud:
# results-bucket: tmp_flank
record-video: true
timeout: 30m
async: false
# project: delta-essence-114723
# results-history-name: ios-history
# test and xctestrun-file are the only required args
test: ./src/test/kotlin/ftl/fixtures/tmp/EarlGreyExample.zip
xctestrun-file: ./src/test/kotlin/ftl/fixtures/tmp/EarlGreyExampleSwiftTests_iphoneos12.1-arm64e.xctestrun
xcode-version: 9.2
device:
- model: iphone8
version: 11.2
locale: en
orientation: portrait
flank:
# test shards - the amount of groups to split the test suite into
# set to -1 to use one shard per test.
testShards: 1
# repeat tests - the amount of times to run the tests.
# 1 runs the tests once. 10 runs all the tests 10x
repeatTests: 1
# always run - these tests are inserted at the beginning of every shard
# useful if you need to grant permissions or login before other tests run
test-targets-always-run:
- a/testGrantPermissions
# test targets - a list of tests to run. omit to run all tests.
test-targets:
- b/testBasicSelection
Run test_runner/flank.yml
with flank to verify Android execution is working.
cd test_runner/
./gradlew clean assemble fatJar
java -jar ./build/libs/flank-*.jar firebase test android run
# gcloud args match the official gcloud cli
# https://cloud.google.com/sdk/gcloud/reference/firebase/test/android/run
gcloud:
results-bucket: tmp_flank
record-video: true
timeout: 30m
async: false
project: delta-essence-114723
results-history-name: android-history
# test and app are the only required args
app: ../test_app/apks/app-debug.apk
test: ../test_app/apks/app-debug-androidTest.apk
auto-google-login: true
use-orchestrator: true
environment-variables:
clearPackageData: true
directories-to-pull:
- /sdcard/screenshots
performance-metrics: true
test-targets:
# supported are class, notClass, size, annotation, notAnnotation, package, notPackage, testFile, notTestFile
# as described in https://developer.android.com/reference/android/support/test/runner/AndroidJUnitRunner
- class com.example.app.ExampleUiTest#testPasses
device:
- model: NexusLowRes
version: 28
flank:
# test shards - the amount of groups to split the test suite into
# set to -1 to use one shard per test.
testShards: 1
# repeat tests - the amount of times to run the tests.
# 1 runs the tests once. 10 runs all the tests 10x
repeatTests: 1
# always run - these tests are inserted at the beginning of every shard
# useful if you need to grant permissions or login before other tests run
test-targets-always-run:
- class com.example.app.ExampleUiTest#testPasses
Download Flank from GitHub releases.
Stable
wget --quiet https://github.com/TestArmada/flank/releases/download/v3.0.0/flank.jar -O ./flank.jar
java -jar ./flank.jar android run
Snapshot (published after every commit)
wget --quiet https://github.com/TestArmada/flank/releases/download/flank_snapshot/flank.jar -O ./flank.jar
java -jar ./flank.jar android run
In CI, it may be useful to generate the file via a shell script:
cat << 'EOF' > ./flank.yml
gcloud:
app: ../../test_app/apks/app-debug.apk
test: ../../test_app/apks/app-debug-androidTest.apk
EOF
Fladle is a Gradle plugin for Flank that provides DSL configuration and task based execution.
Follow the test lab docs to create a service account.
- Save the credential to
$HOME/.config/gcloud/application_default_credentials.json
or setGOOGLE_APPLICATION_CREDENTIALS
when using a custom path. - Set the project id in flank.yml or set the
GOOGLE_CLOUD_PROJECT
environment variable.
For continuous integration, base64 encode the credential as GCLOUD_KEY
. Then write the file using a shell script. Note that gcloud CLI does not need to be installed. Flank works without any dependency on gcloud CLI.
Encode JSON locally.
base64 -i "$HOME/.config/gcloud/application_default_credentials.json" | pbcopy
Then in CI decode the JSON.
GCLOUD_DIR="$HOME/.config/gcloud/"
mkdir -p "$GCLOUD_DIR"
echo "$GCLOUD_KEY" | base64 --decode > "$GCLOUD_DIR/application_default_credentials.json"
flank.yml is compatible with the gcloud CLI.
gcloud firebase test android run flank.yml:gcloud
gcloud alpha firebase test ios run flank.ios.yml:gcloud
NOTE: You will need to activate gcloud's service account for the above commands to work.
Use the doctor command to check for errors in the YAML.
flank firebase test android doctor
flank firebase test ios doctor
Flank supports printing the current version.
$ flank -v
v3.0-SNAPSHOT
Access Not Configured. Cloud Tool Results API has not been used in project 764086051850 before or it is disabled.
This error means authentication hasn't been setup properly. See Authenticate with a service account
in this readme.
How do I use Flank without typing long commands?
Add Flank's bash helper folder to your $PATH environment variable. This will allow you to call the shell scripts in that helper folder from anywhere.
With the flank shell script, you can use flank
instead of java -jar flank.jar
. Examples:
flank android run
flank ios run
With the update_flank.sh shell script, you can rebuild flank.jar
.
Symbol is declared in module 'java.xml' which does not export package 'com.sun.org.apache.xerces.internal.dom'
Make sure you're using JDK 8 to compile Flank.
Test run failed to complete. Expected 786 tests, received 660
Try setting use-orchestrator: false
. Parameterized tests are not compatible with orchestrator. Flank uses orchestrator by default on Android.