Flank is a massively parallel Android and iOS test runner for Firebase Test Lab.
Flank is YAML compatible with the gcloud CLI. Flank provides extra features to accelerate velocity and increase quality.
:speedboat: Massively parallel Android and iOS test runner for Firebase Test Lab
Home Page: https://firebase.community/
License: Apache License 2.0
Flank is a massively parallel Android and iOS test runner for Firebase Test Lab.
Flank is YAML compatible with the gcloud CLI. Flank provides extra features to accelerate velocity and increase quality.
I have some tests used by fastlane+screengrab which should not be ran by flank.
I would like to have a skip-tests
in the config file specifying the package of the tests which shouldn't be run. What do you think?
Also, is there a reason why the package which should be run are not part of the config file instead of the command line args?
Thanks!
The local results
folder is empty. There's also no link to the corresponding Firebase test run.
$ java -jar flank.jar app.apk test.apk com.example.ui.DeviceTest
No Local 'flank.tests' found. It's used to create shards with configurable durations.
Creating bucket: gs://apk-bucket-2017-05-25-12-35
Uploading: app.apk to bucket: gs://apk-bucket-2017-05-25-12-35
Uploading: test.apk to bucket: gs://apk-bucket-2017-05-25-12-35
2 shards will be executed on: Nexus6P
Executing shard 0: com.example.ui.DeviceTest#displaysPageObjects
Executing shard 1: com.example.ui.DeviceTest#checkAnimationsOff
Fetching results to: ~/code/tmp/results
Combined test execution time: 0 seconds
End time: 2017-05-25-12-35
$ gsutil mb gs://delta-essence-114723-flank/
Creating gs://delta-essence-114723-flank/...
ServiceException: 409 Bucket delta-essence-114723-flank already exists.
We should probably check to see if the bucket exists before running the make bucket command.
In my setup, I will have a different cfg for ci and a different cfg for regression testing. Example, I want ci to run in a single locale / single device.
What do you think of removing all the parameters we have right now and force a config.
so instead of:
java -jar Flank.jar <app-apk> <test-apk> [package-name]
it would be:
java -jar Flank.jar myConfig.properties
properties file should have new keys:
app_apk_path
, tests_apk_path
, use_tests_in_packages
to map the previous parameters
With debug = false, all the gcloud commands can fail and flank will happily report success. We should probably check to see if there's anything in the error stream, and if so end the process with a non-zero exit code.
At current state, Flank supports running tests based on:
It's possible to filter tests in such a way passing the package parameter when invoking Flank.
In addition to that is possible to run all but some tests defining the skip-tests
property in the config file.
Taking inspiration from what AndroidJunitRunner supports, we could add these execution options:
Related to #89 (comment), since it would require a new CLI arguments mechanism.
This would be a helpful configuration option that many tools provide, and would allow for more easily performing certain logic in CI environments if tests fail (without globbing and grepping xml files). Thanks for your consideration!
I'd expect the test run to stop on an exception, not proceed with an invalid configuration.
$ java -jar flank.jar app.apk test.apk com.instructure.teacher.ui.DeviceTest
java.lang.NumberFormatException: For input string: "25m"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at com.walmart.otto.configurator.ConfigReader.setProperty(ConfigReader.java:64)
at com.walmart.otto.configurator.ConfigReader.setProperties(ConfigReader.java:23)
at com.walmart.otto.configurator.ConfigReader.<init>(ConfigReader.java:14)
at com.walmart.otto.Flank.main(Flank.java:26)
Local 'flank.tests' found. It contains test execution times used to create shards with configurable durations. Default shard duration is 120 seconds.
Creating bucket: gs://apk-bucket-2017-05-26-11-34
Creating gs://apk-bucket-2017-05-26-11-34/...
Uploading: app.apk to bucket: gs://apk-bucket-2017-05-26-11-34
$ cat config.properties
debug-prints:true
deviceIds: NexusLowRes
os-version-ids: 25
orientations: portrait
locales: en
shard-timeout: 25m
numShards: -1
Instead of creating one shard per test case Flank can create shards that contain tests that add up to a given duration
Is it not possible to have one shard per test case? In terms of fault isolation/video/logs, there are some use cases where this is desirable.
Gradle builds print time info:
BUILD SUCCESSFUL
[22:54:52]: โธ Total time: 4 mins 58.471 secs
It'd be nice to have a Total time
that represents the amount of time from the start of flank until the end. Currently I have to wrap flank in a script to figure out how long we're waiting for flank to run the tests.
Was trying to test Flank but can't get it to execute correctly.
Running command as follows:
java -jar Flank-1.3.0.jar app\build\outputs\apk\app-debug.apk app\build\outputs\apk\app-debug-androidTest.apk com.amazonaws.devicefarm.android.referenceapp
Am greeted with this error message. Note gcloud command is working fine for me when used outside of Flank.
java.io.IOException: Cannot run program "gcloud": CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(Unknown Source)
at com.walmart.otto.tools.ProcessBuilder.start(ProcessBuilder.java:24)
at com.walmart.otto.tools.ProcessExecutor.executeCommand(ProcessExecutor.java:31)
at com.walmart.otto.tools.Tool.executeCommand(Tool.java:69)
at com.walmart.otto.tools.GcloudTool.getProjectName(GcloudTool.java:118)
at com.walmart.otto.Flank.getProjectName(Flank.java:153)
at com.walmart.otto.Flank.start(Flank.java:40)
at com.walmart.otto.Flank.main(Flank.java:73)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 8 more
I have a configuration like this:
deviceIds=Nexus6
os-version-ids=25
orientations=portrait
locales=pt_BR
shard-timeout=60
shard-duration=2400
numShards=8
shardIndex=
debug-prints=true
fetch-xml-files=false
gcloud-path=
gsutil-path=
and two valid apks (just ran the tests with them on my machine and all passed), but when trying to use them to shard with flank, a crash happens:
Exception in thread "main" java.util.EmptyStackException
at java.util.Stack.peek(Stack.java:102)
at java.util.Stack.pop(Stack.java:84)
at com.walmart.otto.shards.ShardCreator.addToShard(ShardCreator.java:56)
at com.walmart.otto.shards.ShardCreator.getShards(ShardCreator.java:39)
at com.walmart.otto.shards.ShardExecutor.executeShards(ShardExecutor.java:43)
at com.walmart.otto.shards.ShardExecutor.execute(ShardExecutor.java:32)
at com.walmart.otto.Flank.main(Flank.java:48)
Note that the test APK has near 900 instrumentation tests (don't know if that can contribute to the problem somehow)
Coverage is fetched as bucket artifact, but it can take significant time to fetch all the artifacts just to get coverage results. It'll save some build time if coverage is fetched along with test_results
End time: 2017-05-25-12-35
Is this a bug? ^
We'd love to be able to specify a json file of shards and have Flank use that. Motivation:
This could be configured via a shardManifest=path/to/shards.json
in the config.properties. Example test JSON file:
[
[
"com.example.android.app.androidapp.ApplicationTest",
"com.example.android.app.view.activity.LoginFlowTest",
"com.example.android.app.view.activity.LogoutFlowTest"
],
[
"com.example.android.app.view.activity.SplashScreenTimeoutTest",
"com.example.android.app.view.activity.HomeFlowTest"
]
]
If you would be interested in this feature, I am happy to attempt a pull request or work with an existing flank contributor to do so :)
Google Cloud SDK 167: "Promoted Firebase Test Lab to beta for running instrumentation test with Android Test Orchestrator. Use --use-orchestrator to run with Orchestrator."
This "allows you to run each of your app's tests within its own invocation of Instrumentation" and avoid shared state and crashes taking out other tests.
Would it be possible to have a configuration option for enabling this?
Version: 1.7.0
java.lang.NullPointerException
at com.walmart.otto.reporter.PriceReporter.getBillableTime(PriceReporter.java:57)
at com.walmart.otto.reporter.PriceReporter.getTotalBillableTime(PriceReporter.java:40)
at com.walmart.otto.Flank.printEstimates(Flank.java:148)
at com.walmart.otto.Flank.start(Flank.java:83)
at com.walmart.otto.Flank.main(Flank.java:107)
Could we make it so the build doesn't fail and the billable time was just left unknown?
When set config.properties
like deviceIds=Nexus5X,Nexus6P os-version-ids=23,24 orientations=portrait locales=en,sv
The cost on Jenkins will looks like 00:41:10 Billable time: 0.0 min(s) 00:41:10 00:41:10 Estimated cost:
It looks like there's no option to retry failed tests? That'd be helpful when 1 test runs in each VM (shard-duration: -1
).
Currently the API jars are sourced from maven or the Android Studio repo. The gcloud SDK generates APIs using google/apitools. Using that same approach for Flank would allow us to generate an always up to date Java binding.
def GenerateApi
in /googlecloudsdk/api_lib/regen/generate.py#L52
gcloud-cli/tools/regen_apis/
The new generation tool is googleapis/toolkit however I haven't found the protobufs required to make that work for the testing API.
ATSL doesn't collect code coverage properly when orchestrator is enabled. A work around is to disable orchestrator. The ATSL team may fix this issue in an upcoming release.
https://issuetracker.google.com/issues/72758547
As Firebase test lab lacks a public issue tracker, I'll track the known issues here since they also impact Flank.
Why is it necessary to upload the apks to a gs bucket instead of using the ones on the provided path directly?
Another thing, are there plans to support other kind of buckets? My current CI saves the apks on a s3 bucket on aws, it would be nice to fetch than from there and not needing to replicate them to gs://<>
The last thing, is there a CONTRIBUTING.md
guide somewhere or is just to fork and make a PR with the desired feature?
The IntelliJ linter is picking up some weird Java issues. We should probably fix them. And then add linting to the Travis CI build.
public String uploadAPKsToBucket() {
SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd-hh-mm-ss");
This will break the moment two PRs are submitted at the same time. This isn't unusual with a large team. If you checkout the Google cloud Python SDK code, they add some random numbers in addition to the timestamp.
Due to changes in naming convention on FTL side, after fetching results there is only one file test_results_0.xml
and it corresponds to tests executed on the last shard. The same is also applied to coverage.ec
.
We have two projects where we use Flank, one of them works fine, but another fails to fetch test run results. Projects have identical Flank configuration except gcloud-bucket
.
After all tests pass Flank crashes with following log:
Created folder: /project/instrumentation-tmp/results
Fetching results to: /project/instrumentation-tmp/results
java.lang.RuntimeException: CommandException: No URLs matched: flank.tests
at com.walmart.otto.tools.Tool.executeCommand(Tool.java:60)
at com.walmart.otto.tools.GsutilTool.uploadTestTimeFile(GsutilTool.java:81)
at com.walmart.otto.Flank.uploadTestTimeFile(Flank.java:151)
at com.walmart.otto.Flank.start(Flank.java:65)
at com.walmart.otto.Flank.main(Flank.java:77)
Environment:
I've also double checked gcloud-bucket
and gcloud --key-file
, it seems to be a Flank issue since it's able to run tests but unable to download results.
java -jar flank.jar app.apk test.apk
Usage: Flank
I'd expect ^ to run all tests. Instead I have to specify a package. Thoughts on allowing all tests to run when the package is omitted? This is more in line with the gcloud CLI. Package is optional and default is to run everything.
When flank generated the shards, firebase gave a 400
saying that the test is invalid.
This gist has the complete log.
The apks (with I already sent to @renas, if anyone want to test) run successfully on my machine.
It was also a bit hard to understand what logs are from what shard. Maybe putting a common prefix to logs about what shard they're refering to could help debugging.
E.g:
https://gist.github.com/thalescm/7e64bc80fc2a32dcb8eef50fc0a96aac#file-flank-shard-error-L51-L56
all lines could start with something like SHARD X: <real log>
Request: Add git tags for versions
For example, ideally there would be a git tag for 1.4.1
TL;TR: we want sometimes run our instrumentation tests in Firebase, but most of the time run them on internal emulators pool.
So we're completely fine with free plan quotas but Flank requires paid plan (for Google Cloud Storage bucket I guess).
What is the reason to require paid plan and can this be fixed to allow test runs on free plan? Thanks!
Can't process test artifacts for: testSurfaceLogging
Can't process test artifacts for: testAppQuitViaActivityFinish
Can't process test artifacts for: testSaveObstruction
Can't process test artifacts for: testOverflowObstruction
Can't process test artifacts for: testTriedItEndsImpressions
Can't process test artifacts for: testSaveObstruction
Can't process test artifacts for: testSaveObstruction
java.lang.RuntimeException: java.lang.RuntimeException: java.nio.file.NoSuchFileException: /project/bucket/2018-01-24_18:01:44.378000_Mlhf/41/NexusLowRes-25-en-portrait/logcat
The xml aggregated and uploaded fine. I, unfortunately, don't know if the not found file was ever created. I believe that the can't process test artifacts were all shards that partially failed.
As you can do with gcloud
command, you can specify device ids and versions, or you can specify each device with it's own config (version, orientation, locale and model)
The point is to add support for this type of configuration as well on config.properties
so you could have a specified device with API 19 and another with API 25.
Today when trying to do this you would end up with four configurations (both devices with API 19 and 25) instead of two.
It looks like the only test flank has is empty?
It'd be nice to setup some E2E tests on Travis CI using encrypted variables.
While running tests in FTL with and without Flank i see some tests not running in Flank case:
seems all of them has @RunWith(Parameterized::class)
in common.
Is it possilble to support them?
renas commented on f074037 a minute ago
@bootstraponline I'm getting this error now when I run test suites containing large amount of tests:Uploading [gs://walmart-fb668-flank/2017-06-12_17:52:57.388000_ShhK/walmart-android-qa-debug.apk] to Firebase Test Lab...
ERROR: (gcloud.firebase.test.android.run) Could not copy [gs://walmart-fb668-flank/2017-06-12_17:52:57.388000_ShhK/walmart-android-qa-debug.apk] to [gs://walmart-fb668-flank/2017-06-12_17:52:57.388000_ShhK/] ResponseError 429: The total number of changes to the object walmart-fb668-flank/2017-06-12_17:52:57.388000_ShhK/walmart-android-qa-debug.apk exceeds the rate limit. Please reduce the rate of create, update, and delete requests..Before when we only had a root project we didn't experience this error. Seems to have come with the sub-folder object that we now create.
moving this to an issue so it can be tracked properly.
CI helps make sure that PRs don't break the code. I've submitted a .travis.yml integration. Someone with admin rights will have to enable Travis for this repo. After that, a badge can be added to the readme.
I have a test suite with 112 tests. These tests run on Firebase test lab on every pull request. Currently, we're using a single VM. The configuration is defined in fastlane. Our apps, tests, and build automation are all open source.
gcloud test runs directly on a single VM and takes 0:09:39 (112 tests).
flank with 112 VMs takes 0:13:24.
Flank increased our cost by 18x and is 3m 45s slower.
$ cat config.properties
debug-prints:true
deviceIds: NexusLowRes
os-version-ids: 25
orientations: portrait
locales: en
shard-timeout: 5
shard-duration: -1
The stdout is filled with / [102/113 files][ 2.0 MiB/ 2.1 MiB] 93% Done
. It looks like more commands require the quiet flag. The performance loss is because we're:
flank
bucketIn the 1 test per VM model, everything we need is already in stdout. The test name, success/failure and a link to the full report. Skipping XML fetching would be a huge speed win.
In terms of avoiding spinning up the processes, you can check out this proof of concept:
I'm spinning up multiple test VMs from a single python process by extending the SDK with a new shard
command.
Combined test execution time: 26 seconds
seconds are round up to minutes by Firebase test lab
thoughts on having flank output estimated cost for the test run? The combined test execution time isn't sufficient to figure out the expense. The output would look something like this:
Estimated cost: $0.033 (virtual), $0.166 (physical)
2 tests at <1 min each
1/60.0 * 2 = $0.033
virtual device5/60.0 * 2 = $0.166
physical deviceThe timing info in the results.xml files is wrong. Firebase is aware of the issue but they probably won't fix it anytime soon. I think overriding the xml files with the actual time as recorded by flank might be helpful. This would allow, among other things, accurate cost estimation.
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<testsuite errors="0" failures="0" hostname="localhost" name="" skipped="0" tests="2" time="1.823" timestamp="2017-05-26T15:40:36">
<properties/>
<testcase classname="com.instructure.teacher.ui.DeviceTest" name="checkAnimationsOff [NexusLowRes-25-en-portrait]" time="0.0"/>
<testcase classname="com.instructure.teacher.ui.DeviceTest" name="displaysPageObjects [NexusLowRes-25-en-portrait]" time="0.0"/>
</testsuite>
The issue is time="0.0"
which isn't true.
Hello,
Thanks for the great project and open-sourcing it. I have been successfully able to run our entire suite of ~60 tests with minimal effort locally and also on our Jenkins.
I am observing a rather strange behavior: All the tests though executed in a separate matrix are essentially showing the same artifacts including logcat/video.mp4 file. In the gsutil pull down command I can clearly see only one video file being generated.
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/test_result_7874888483897436532.xml...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/test_result_9131515297737460228.xml...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/test_result_8667431412748016253.xml...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/test_result_8330274942631895392.xml...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/test_result_9101555559323915915.xml...
[android] Operation completed over 49 objects/16.2 KiB.
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/artifacts/com.godaddy.gdm.telephony.uitests.SwitchAccountNumberTest-switchAccountTest-switchAccountTest-1.jpg...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/artifacts/video.mp4...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/artifacts/coverage.ec...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/bugreport.txt...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/logcat...
[android] Copying gs://gdm-android-telephony-flank/2017-07-24_21:37:29.481000_AdAA/Nexus6P-25-en-portrait/instrumentation.results...
When I open the test_results file I can clearly see that tests are being executed and some have proper failure logs with them. But I can't find the test being executed in logical. Is there a setting I am missing?
My config.properties look pretty normal
deviceIds=Nexus6P
os-version-ids=25
orientations=portrait
locales=en
environment-variables=coverage=true,coverageFile=/sdcard/coverage.ec
directories-to-pull=/sdcard
shard-timeout=5
shard-duration=-1
numShards=
shardIndex=
debug-prints=true
fetch-xml-files=true
fetch-bucket=false
gcloud-path=
gsutil-path=
do let me know what details you need to diagnose this problem.
Right now it's a bit of a pain to have to update the flank.tests every time a new test is added. If new tests would automatically be added to other shards based on an average test duration time, or a default value specified in config.properties, it would save a bit of time having to update the test times every time a new test is added.
It looks like I'm not able to move cards across the board. ๐ญ
java -jar Flank-1.4.1.jar walmart-android-app-dev-debug.apk walmart-android-app-dev-debug-androidTest.apk
Downloading 'flank.tests'. It contains test execution times used to create shards with configurable durations. Default shard duration is 120 seconds.
java.lang.StringIndexOutOfBoundsException: String index out of range: 6
at java.lang.String.substring(String.java:1963)
at com.walmart.otto.tools.GsutilTool.uniqueObjectName(GsutilTool.java:33)
at com.walmart.otto.tools.GsutilTool.uploadAPKsToBucket(GsutilTool.java:64)
at com.walmart.otto.Flank.start(Flank.java:59)
at com.walmart.otto.Flank.main(Flank.java:77)
I've recently upgraded to flank 1.5.0 and started seeing the following output
Executing shard 107: com.godaddy.gdm.telephony.WebSocketTest#testIncomingSmsWebSocket
Fetching bucket to: /Users/smalugu/code/gdm-android-telephony/bucket
Billable time: 0.0 min(s)
Estimated cost:
[2017-12-02_09:13] Total time: 0 min, 25 sec
When I check google firebase console, I see no tests. If I change to flank 1.4.0 tests run just fine.
My config.properties look like this
deviceIds=NexusLowRes
os-version-ids=25
orientations=portrait
locales=en
environment-variables=^:^coverage=true:coverageFile=\"/sdcard/coverage.ec\":notAnnotation=android.support.test.filters.FlakyTest,com.godaddy.gdm.telephony.uitests.testcategories.DeviceOnlyTests
directories-to-pull=/sdcard
shard-timeout=5
shard-duration=-1
numShards=
shardIndex=
debug-prints=false
fetch-xml-files=false
fetch-bucket=true
gcloud-path=
gsutil-path=
I also tried changing to new syntax for devices and that didn't help. Is there something I am missing?
You can read more about this in #89
Requirements:
java -jar Flank.jar -a <app-apk> -t <test-apk> -c <config-path> -p [package-name]
a
, t
are required, c
and p
are optional.
We had to @Ignore
couple tests and that turned out to be problem for Flank, because it tries to run them anyways (by passing -e class class1#test1,class1#test2,class2#test1
) and that crashes android.support.test.runner.AndroidJUnitRunner
with following stacktrace:
java.lang.NullPointerException: Attempt to invoke interface method 'int java.lang.CharSequence.length()' on a null object reference
at java.util.regex.Matcher.reset(Matcher.java:995)
at java.util.regex.Matcher.<init>(Matcher.java:174)
at java.util.regex.Pattern.matcher(Pattern.java:1006)
at android.support.test.internal.runner.TestRequestBuilder$MethodFilter.stripParameterizedSuffix(TestRequestBuilder.java:523)
at android.support.test.internal.runner.TestRequestBuilder$MethodFilter.evaluateTest(TestRequestBuilder.java:509)
at android.support.test.internal.runner.TestRequestBuilder$ParentFilter.shouldRun(TestRequestBuilder.java:145)
at android.support.test.internal.runner.TestRequestBuilder$ClassAndMethodFilter.evaluateTest(TestRequestBuilder.java:451)
at android.support.test.internal.runner.TestRequestBuilder$ParentFilter.shouldRun(TestRequestBuilder.java:145)
at org.junit.runner.manipulation.Filter$3.shouldRun(Filter.java:112)
at org.junit.runners.ParentRunner.shouldRun(ParentRunner.java:434)
at org.junit.runners.ParentRunner.filter(ParentRunner.java:382)
at org.junit.runner.manipulation.Filter.apply(Filter.java:97)
at android.support.test.internal.runner.TestRequestBuilder$LenientFilterRequest.getRunner(TestRequestBuilder.java:412)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:58)
at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:369)
at
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1962)
And of course Firebase marks test run as successful even though no tests were run.
There was added support of passing the argument --no-record-video
. It would be great to pass it with flank to get faster test results
I am trying out flank for the first time.
What should be my guideline?
In the Medium article about flank (https://medium.com/walmartlabs/flank-smart-test-runner-for-firebase-cf65e1b1eca7) a paragraph describes that flank should output a JUnit report as if all tests were run on a single device.
However when I'm running all tests with flank I'm only getting all the *.xml
reports for the testsuite and no combined report.
Is there a flag that I'm missing to generate this report? Or what is the preferred way to combine all the XML reports to one unified HTML report?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.