The problem
Our current config looks like this:
apiVersion: v1
metadata:
name: Feature XYZ
tags:
- e2e
- release team
- other tag
build: Release $CI_COMMIT_SHORT_SHA
capabilities:
- browserName: "chrome"
- browserName: "firefox"
files:
- ./tests/example.test.js
image:
base: saucelabs/stt-playwright-jest-node
version: v0.1.3
sauce:
region: us-west-1
With the current config, we can specify test file (patterns) and folders, which then would run against the specified capabilities (effectively, the tests get multiplied against the capabilities).
This approach has two limitations:
- If the test folder contains dependencies that are shared across different test files, the only easy way to include them, is to specify the entire parent folder. However, this would cause all tests in that folder to be executed and thus comes with the loss of fine grained control over the test execution. The alternative for the user is to not have any shared dependencies in the tests folder at all (i.e. write test code that doesn't require them or move dependencies out as NPM packages that could then be installed before the test run, a feature that we do not yet support, but users are always welcome to use their own image that comes with pre-installed dependencies in the meantime).
- Since all test files get multiplied against each capability, it is not possible to have only a subset of tests run against a certain capability (e.g. to save on execution time or to target specific capabilities with specific tests). This would require the user to manage multiple config files, which is a little arduous, but could also be improved by enhancing saucectl to pickup multiple configs.
To address at least one of these limitations, I'd like us to explore the following config variants.
Variant 1
apiVersion: v1
metadata:
name: Feature XYZ
tags:
- e2e
- release team
- other tag
build: Release $CI_COMMIT_SHORT_SHA
capabilities:
- browserName: chrome
- browserName: firefox
test:
root: ./tests/
match: ["*.spec.js"]
image:
base: saucelabs/stt-playwright-jest-node
version: v0.1.3
sauce:
region: us-west-1
The root context folder allows the user to specify the entire test folder structure that includes any dependencies. However, with the match
property, the user can control which test files should be executed.
Each matched test file results in a new sauce job.
The user would have to manage multiple configs if they'd like a different set of tests.
Variant 1 therefore addresses the first limitation, but not the second.
Variant 2
apiVersion: v1
metadata:
name: Feature XYZ
tags:
- e2e
- release team
- other tag
build: Release $CI_COMMIT_SHORT_SHA
test:
- name: "chrome"
capabilities:
browserName: "chrome"
root: ./tests/
match: ["*.spec.js"]
- name: "firefox"
capabilities:
browserName: "firefox"
root: ./tests/
match: ["*.spec.js"]
image:
base: saucelabs/stt-playwright-jest-node
version: v0.1.3
sauce:
region: us-west-1
The root
and match
additions are identical to Variant 1. Each matched test file results in a new sauce job. However, different sets of tests can now be specified and the capabilities have moved closer to each individual set.
Test executions will now not be multiplied, but instead run against a single capability defined in the test set.
If the user wants to run the same or different tests against the same or different capabilities, they'd have to enumerate those as a list, which can be seen in the example above.
Variant 2 therefore addresses both limitations that are present in the current config.
Variant 3
apiVersion: v1
metadata:
name: Feature XYZ
tags:
- e2e
- release team
- other tag
build: Release $CI_COMMIT_SHORT_SHA
test:
- name: "the likely default"
capabilities:
- browserName: "chrome"
- browserName: "firefox"
root: ./tests/
match: ["*.spec.js"]
- name: "living on the edge"
capabilities:
- browserName: "edge"
root: ./tests/
match: ["edgy.set.of.tests.spec.js"]
image:
base: saucelabs/stt-playwright-jest-node
version: v0.1.3
sauce:
region: us-west-1
This variant builds on top of the changes of Variant 2, but allows for a list of capabilities to be specified per test set.
The test set is run against each capability. Each matched test file results in a new sauce job.
The only difference between Variant 2 and Variant 3, is that in the former, the user has to create a new set for each capability. The latter allows multiple capabilities to be defined with the same set of tests.
The result between the two variants is identical and only differs in what is enumerated.