A command line interface for the Sauce Labs platform.
License: Apache License 2.0
The saucectl command line interface orchestrates the tests in your framework, providing rich parallelization,
test history filtering, and analytics in Sauce Labs.
For information on how to contribute to saucectl please have a look into our contribution guidelines.
saucectl is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
A per-suite retry logic (and an associated per-suite timeout enforcement) at the runner level to allow automatic retry of parts of a concurrent (>1) run of saucectl.
User Experience enhancement moving logic implementation of retries from users to tool configuration.
In some cases, with either existing or in-development test suites, users may face test flakiness due to infrastructure inconsistencies (read: hardware devices and cosmic rays - no this isn't a joke). In such an event, specifically when using saucectl with concurrency: >1, facing test failures could lead to a re-run of test suites that may have been successful, wasting resource allocation that could have been used for other test runs.
Thus, the request would be in these parts:
--suiteTimeout enforced by saucectl as a per-suite timeout limit, instead of a full saucectl timeout limit, working in tandem with--suiteAutoRetry [true, false] and --suiteMaxRetries X implementing saucectl-driven test suite retry logic, simplifying test retries and limiting retries to failed suites instead of users having to write parsing logic to deduce - or just re-run the saucectl invoke outright.As a developer I want to debug the browser as if it would run without using Docker. Therefor it is essential that it is possible to attach a local Chrome DevTools app to the running browser.
Workflow:
Test executes correctly and pass, but something happens that assets are not uploaded in saucelabs.
saucectl version 0.12.1
(build cfd517f)
using mac
The test passes but I'm getting this final error, so when I go to saucelabs it doesn't show information nor logs of what happened
To reproduce I used the official repo example
$ saucectl new
I chose Cypress JS and I run the test without editing any file with
$ saucectl run
Preparing assets for cypress/integration/example.test.js
upload failed: TypeError: Cannot read property 'startsWith' of null
at SauceLabs._uploadJobAssets (/home/seluser/node_modules/saucelabs/build/index.js:296:64)
at exports.sauceReporter (/home/seluser/src/sauce-reporter.js:94:9)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async report (/home/seluser/src/cypress-runner.js:56:5)
at async cypressRunner (/home/seluser/src/cypress-runner.js:118:20)
at async /home/seluser/bin/cypress:4:3
Open job details page: https://app.saucelabs.com/tests/9abb4787592d4def97cfad3943461c90
https://app.saucelabs.com/tests/9abb4787592d4def97cfad3943461c90
Problem:-
An error saying "No tests found, exiting with code 1" is coming when i execute saucectl run command.
Environment:-
Windows 10
saucectl version 0.15.0
CONFIG.YML
apiVersion: v1alpha
metadata:
name: Feature XYZ
tags:
- e2e
- release team
- other tag
build: Release $CI_COMMIT_SHORT_SHA
files:
Details
I am very new to saucectl and trying to explore and i have been following the below documentation link
https://github.com/saucelabs/testrunner-toolkit
Logs
PS C:\sauceLab_PuppeteerDemo> saucectl run
[email protected] test /home/seluser
DISPLAY="$(cat DISPLAY)" DEBUG="puppeteer:*" jest --config=./.config/jest.config.js --runInBand
cat: DISPLAY: No such file or directory
No tests found, exiting with code 1
Run with --passWithNoTests to exit with code 0
In /home/seluser
10 files checked.
testMatch: /home/seluser/tests/example.test.js - 0 matches
testPathIgnorePatterns: /node_modules/ - 10 matches
testRegex: - 0 matches
Pattern: - 0 matches
npm ERR! Test failed. See above for more details.
If a Docker image has a typo saucectl fails with:
7:48PM ERR Failed to setup test environment error="Error response from daemon: pull access denied for saucelabs/stt-puppeteer-node, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"which doesn't really help understand the reason for the error. I suggest to check for the organisation name of the docker image and have a better wording for this if the image is one of our own maintained images.
If I install saucectl via:
$ npm install -g saucectlI expect the signup command to work but executing it fails:
$ saucectl signup
Some main description
Usage:
saucectl [command]
Available Commands:
configure Configure your Sauce Labs credentials
help Help about any command
new Start a new project
run Run a test on Sauce Labs
Flags:
-h, --help help for saucectl
--verbose turn on verbose logging
-v, --version print version
Use "saucectl [command] --help" for more information about a command.Our current config looks like this:
apiVersion: v1
metadata:
name: Feature XYZ
tags:
- e2e
- release team
- other tag
build: Release $CI_COMMIT_SHORT_SHA
capabilities:
- browserName: "chrome"
- browserName: "firefox"
files:
- ./tests/example.test.js
image:
base: saucelabs/stt-playwright-jest-node
version: v0.1.3
sauce:
region: us-west-1
With the current config, we can specify test file (patterns) and folders, which then would run against the specified capabilities (effectively, the tests get multiplied against the capabilities).
This approach has two limitations:
To address at least one of these limitations, I'd like us to explore the following config variants.
apiVersion: v1
metadata:
name: Feature XYZ
tags:
- e2e
- release team
- other tag
build: Release $CI_COMMIT_SHORT_SHA
capabilities:
- browserName: chrome
- browserName: firefox
test:
root: ./tests/
match: ["*.spec.js"]
image:
base: saucelabs/stt-playwright-jest-node
version: v0.1.3
sauce:
region: us-west-1
The root context folder allows the user to specify the entire test folder structure that includes any dependencies. However, with the match property, the user can control which test files should be executed.
Each matched test file results in a new sauce job.
The user would have to manage multiple configs if they'd like a different set of tests.
Variant 1 therefore addresses the first limitation, but not the second.
apiVersion: v1
metadata:
name: Feature XYZ
tags:
- e2e
- release team
- other tag
build: Release $CI_COMMIT_SHORT_SHA
test:
- name: "chrome"
capabilities:
browserName: "chrome"
root: ./tests/
match: ["*.spec.js"]
- name: "firefox"
capabilities:
browserName: "firefox"
root: ./tests/
match: ["*.spec.js"]
image:
base: saucelabs/stt-playwright-jest-node
version: v0.1.3
sauce:
region: us-west-1
The root and match additions are identical to Variant 1. Each matched test file results in a new sauce job. However, different sets of tests can now be specified and the capabilities have moved closer to each individual set.
Test executions will now not be multiplied, but instead run against a single capability defined in the test set.
If the user wants to run the same or different tests against the same or different capabilities, they'd have to enumerate those as a list, which can be seen in the example above.
Variant 2 therefore addresses both limitations that are present in the current config.
apiVersion: v1
metadata:
name: Feature XYZ
tags:
- e2e
- release team
- other tag
build: Release $CI_COMMIT_SHORT_SHA
test:
- name: "the likely default"
capabilities:
- browserName: "chrome"
- browserName: "firefox"
root: ./tests/
match: ["*.spec.js"]
- name: "living on the edge"
capabilities:
- browserName: "edge"
root: ./tests/
match: ["edgy.set.of.tests.spec.js"]
image:
base: saucelabs/stt-playwright-jest-node
version: v0.1.3
sauce:
region: us-west-1
This variant builds on top of the changes of Variant 2, but allows for a list of capabilities to be specified per test set.
The test set is run against each capability. Each matched test file results in a new sauce job.
The only difference between Variant 2 and Variant 3, is that in the former, the user has to create a new set for each capability. The latter allows multiple capabilities to be defined with the same set of tests.
The result between the two variants is identical and only differs in what is enumerated.
When running:
saucectl configureand halfway through filling out the form I press the Escape key I get an error:
SauceLabs username (enriquegh) 12:17:52 ERR failed to execute configure command error="Unexpected Escape Sequence: ['\\x1b' '\\x1b']"
Similar error can be seen by sending a Control + C:
12:23:35 ERR failed to execute configure command error=interrupt
Interestingly enough this does not happen when using kill -2 process_id (i.e. SIGINT)
Gracefully quit when escape key is pressed.
saucectl configuresaucectl --version
saucectl version 0.25.3
(build 2c39a71bcb6c7f31f18e696f4cda70432210eebf)Currently, Testrunner Toolkit has minimal support for Cypress, TestCafe, Playwright, Puppeteer but we do not see any framework that is Ruby language supported. We would want RSpec + Capybara + Selenium support in future.
Many Ruby community would be happy to see the pre-requisites to bring scalable testing earlier into the software lifecycle (shift-left) to speed up delivery and increase quality, making it easier to test earlier
In the testing cycle.
We have various types of configurations when running a test with saucectl. They can be categorized as:
The first two are already part of the current config specification. The last one isn't. Having framework specific configurations is particularly important for frameworks like WebdriverIO that define a configuration for running tests. However this can be also useful for already supported frameworks like Puppeteer or Playwright that run on Jest where the Jest configuration is currently hard coded into the runner images.
After some initial discussions I propose to create a second configuration file that user can apply to their test runs, similar to the Kubernetes model, e.g.:
$ saucectl run -c ./.sauce/main.config.yml -c ./.sauce/framework.config.ymlThe saucectl would then propagate the configurations to the runner where it will be consumed and applied. In order to distinguish the different types of configurations we would introduce a config property called Type that defines the configuration type. We already define this in our current configurations but ignore the value. The format of this property depends on the design we decide to chose.
The configuration should be validated and send over to the runner image. There should be a fixed defined place for all configurations that allows the runner image to consume the configurations as files. I think Yaml is a format that can be read by all languages. Every framework has different kind of configurations. There are two ways how we can handle these:
One options is to allow the runner to define the format of the configurations based on the framework combination it offers (e.g. Jest + Puppeteer or TestNG + Selenium). I can also see there are even more setting variations depending on what the runner supports. This all can be then defined in a single Yaml file.
The general format should be as follows:
apiVersion: v1
# type of object: framework configurations
kind: FrameworkOptions
# maybe high level sauce configurations available in all runners
option1: "foo"
option2: "bar"
# a list of frameworks and their corresponding options
frameworks:
# TestNG specific settings
- TestNG:
option1: "foo"
option2: "bar"
# Selenium specific settings
- Selenium:
option1: "foo"
option2: "bar"saucectl to run and only need to be copied over to the container)Another option could be to have a config file per framework. The structure of that would then look as follows:
jest.config.yaml:
kind: FrameworkOption
spec:
testEnvironment: node
reporters:
- spec
- junitpuppeteer.config.yaml:
kind: FrameworkOption
base: ./puppeteer.main.config.yaml
spec:
headless: true
defaultViewport:
width: 800
height: 600Then run these configs with saucectl via:
$ saucectl run -c main.config.yaml -c jest.config.yaml -c puppeteer.config.yamlpuppeteer.staging.config.yaml and a puppeteer.dev.config.yaml and replace them when calling saucectlIn the JavaScript ecosystem it is common that such configuration files are defined as executable JavaScript files, e.g. Webpack, that allows users to dynamically create a configuration object based on arbitrary conditions (e.g. environment variables). I can see this being also useful for this case particularly given that e.g. WebdriverIO users can define their config as a JS file but need to define it in yaml using saucectl. I would consider two options here:
have a broader discussion about the configuration design that would allow to define the saucectl configuration files as language files (e.g. JS files)
allow to reference a file that is being copied into the Docker container that can be used as config, e.g.
Proposal A:
apiVersion: v1
kind: FrameworkOptions
frameworks:
- WebdriverIO: "/.sauce/wdio.conf.js"Proposal B:
apiVersion: v1
kind: FrameworkOption
spec:
ref: "/.sauce/wdio.conf.js"This would mean no other configs are allowed as we don't want to handle any merge procedures.
If I install saucectl through Node.js via:
$ npm i -g saucectlI expect to see the version and the build sha when running:
$ saucectl -vbut I only get unknown version:
$ saucectl -v
saucectl version unknown-version
(build unknown-commit)The sites we'd like to test via saucectl are all within our vpn. I can't seem to find any documentation for the tool to use the shared tunnels we have setup in our saucelabs connect app.
Is this not supported?
Are there any workarounds?
In particular, we'd like to run e2e tests against staging hosts as a part of our jenkins pipeline using Cypress via saucelabs (sio we can run them in multiple browsers & devices)
Passing env var by cmd line or by sauce config doesn't work.
saucectl run -e foo=bar
or
suites:
- name: "cypress test"
browser: "chrome"
config:
testFiles: [ "**/*.*" ]
env:
foo: bar
Env var should be applied successfully.
saucectl version 0.20.0
(build 8820dbf)
cypress runner: v0.2.2
TestCafe allows me to pass additional arguments for the browser, like -incognito --use-fake-ui-for-media-stream. It would be nice if we could do the same for Sauce Labs running TestCafe. For example:
browserName: "chrome"
browserArgs: ["--no-sandbox", "--disable-features=site-per-process"]
This would be similar to something that already exists on Sauce Labs for Puppeteer.
Allows more flexibility on the testing environment and standardizes the configuration among different testing frameworks.
Also, it could be a deal killer for customers that would require certain features to be mocked using these arguments, like camera or microphone.
For our particular case, we work with audio and video as our core scenario. Currently, we enable the following flags to instruct Google to mock those devices for us: --use-fake-ui-for-media-stream --use-fake-device-for-media-stream
Using SauceCTL + Testcafe + iOS sims fails, it starts a safari browser on OSX, see this execution
Testcafe should run on iOS sims
apiVersion: v1alpha
kind: testcafe
# Sauce Labs specific information
sauce:
region: eu-central-1 # or us-west-1
concurrency: 10 # Controls how many suites are executed at the same time.
metadata:
name: TestCafe on iOS Simulators with SauceCTL
tags:
- testcafe
- ios simulators
build: "TestCafe: iOS Simulator test set - $DATE"
# Testcafe specific info
testcafe:
version: 1.14.2
# Controls what files are available in the context of a test run (unless explicitly excluded by .sauceignore).
rootDir: ./
# Tests
suites:
- name: "Testcafe - iOS Tests - Safari"
browserName: "safari"
src:
# test files glob
- "**/login.*"
simulators:
- name: iPhone 12 Simulator
platformName: iOS
platformVersions:
- "14.3"
- name: iPad mini (5th generation) Simulator
platformName: iOS
platformVersions:
- "14.3"
# Controls what artifacts to fetch when the suite on Sauce Cloud has finished.
artifacts:
download:
directory: ./artifacts
match:
- '*'
when: always 0.52.2
The current behaviour of --env SOME=THING appears to not set environment variables "in" the device when launching a test (as is supported by xcodebuild-run tests). I can see USER, XPC_FLAGS, XPC_SERVICE_NAME and others, but additional --env-provided values seems to be broken.
The expected behaviour is that using saucectl's --env SOME=THING cli option would set environment variables on the target as is being done by xcodebuild-driven testsé
Run any iOS xcuiapplication-based tests that print out the whole environment. Example code to do that in a setup method:
for (NSString* key in [[NSProcessInfo processInfo] environment].allKeys) {
NSLog(@"Full map reveal - key: %@", key);
NSLog(@"Full map reveal - element: %@", [[NSProcessInfo processInfo] environment][key]);
NSLog(@"==================================================");
}
0.59.x
Currently there are various of config files available, e.g. kind: playwright, kind: cypress etc. Is the idea to also allow different kind of Sauce products, e.g. API testing, to be embedded into this concept? If so, how should this look like? I see two options for this:
saucectl where the integration has a small footprint and the actual functionality can be implemented in a separate repository by a separate team owning their own pipelines to deliver featuresWhat is the team thinking about moving towards extending configuration types?
I am working on a new product and @guinaut has mentioned interest to integrate it into saucectl.
I believe this makes totally sense particularly given that users don't only want to run one type of testing but multiple. For that reason we have implemented the kind property that allows to define different types of configurations, e.g.:
$ saucectl run -f cypress.sauce.conf.yaml -f api-tests.conf.yaml -f ...When specifying a package for a test suite, actually all tests from the test apk are run
Only tests from the specified package should run
apiVersion: v1alpha
kind: espresso
sauce:
region: eu-central-1
concurrency: 1
metadata:
name: Espresso Tests
tags:
- e2e
- android
build: "PR: $BITRISE_GIT_MESSAGE"
espresso:
app: ./app/build/outputs/apk/kaufda/debug/kaufdaDebug-21.10.0.D.apk
testApp: ./app/build/outputs/apk/androidTest/kaufda/debug/app-kaufda-debug-androidTest.apk
suites:
- name: "Feature Tests"
testOptions:
package: com.bonial.tests.feature
clearPackageData: true
useTestOrchestrator: true
devices:
- name: "Samsung Galaxy.*"
platformVersion: 9
artifacts:
download:
when: always
match:
- junit.xml
directory: ./artifacts/
saucectl version 0.56.1
(build 65f13cfa0d88f317c870ba100e418552c9fdef36)
Install saucectl and set up a framework:
$ npm i -g saucectl
$ npm configIf Docker is running, close it. Then run a test.
I expect that the testrunner fails with an error message that Docker is not installed or running and provides me with instructions how to get going.
Testrunner just stops with no error message:
$ saucectl run ./.sauce/config.yml
5:56PM INF Start Run Command
5:56PM INF Reading config file config=.sauce/config.yml
5:56PM INF Starting local runnerIt seems it is unrelated to Docker, even with Docker started it has the same behavior. Here is my example project:
Enhancement on the junit results file
User Comfort - not having to correct the junit.xml file to simplified, single-file junit specification.
Currently, the resulting junit.xml result files are not technically completely to junit spec (single-file mode), in that they contain only one testSuite each, without an overarching testSuites element - meaning that a tool expecting testSuites breaks down.
One way to remedy this would be to automatically collate junit.xml files from all the n test runs (from 1 to the maximum abuse your service can take, in theory) into one junit-meta.xml that would automatically collate multiple test suites into one file with a testSuites element.
The simplest illustration is if a user of saucectl would have an existing tool in their toolset that takes an xml with one testSuites element as input, and creates a styled html report as an output.
Our company currently uses SauceLabs for virtual and a few real devices with a moderate concurrent limit. Upon learning about the testrunner after the new year, we downloaded the basic test and decided to try running through the provided, basic test that install alongside saucectl.
After setting up my username and key from both my user account, our admin account into my .bash_profile as well as my .zprofile and confirmed the environment variables through via echo $USERNAME, upload running saucectl run within the root directory, the following error occurs after completion of the test:
Using /home/seluser/assets/example.test.js.mp4 as the main video.
Create job failed: Error: Failed calling createJob: Response code 403 (Forbidden)
at Proxy. (/home/seluser/node_modules/saucelabs/build/index.js:233:15)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
Unable to retrieve test entry. Assets won't be uploaded.
4:38PM INF Command Finished ExitCode=0
4:38PM INF Tearing down environment
In addition, I tried switching from my username in the admin portal to my email address and the issue still failed, but I did get a 401 error when an email was used.
Results are uploaded to saucelabs.
`apiVersion: v1alpha
kind: cypress
sauce:
region: us-west-1
metadata:
name: Testing Cypress Support
tags:
- e2e
- release team
- other tag
build: Release $CI_COMMIT_SHORT_SHA
cypress:
configFile: cypress.json
projectpath: cypress
envfile: ""
suites:
saucectl version 0.22.0 - (build ecdc0ad)
npm version 6.14.10
macOS Catalina version 10.15.7
docker version 20.10.0, build 7287ab3
It would help usability of saucectl if we validated the contents of the config.yml and gave helpful error messages
saucectl run not do anythingSteps to reproduce
git clone [email protected]:dpgraham/testbed.git && cd testbed
npm install
saucectl run --config .sauce/no-files-config.yml
# .sauce/no-files-config.yml
apiVersion: v1alpha
metadata:
name: Feature XYZ
tags:
- e2e
- release team
- other tag
build: Release $CI_COMMIT_SHORT_SHA
suites:
- name: "saucy test"
match: ".*.(spec|test).js$"
image:
base: saucelabs/stt-cypress-mocha-node
version: latest
sauce:
region: us-west-1
sauce-cypress-runner git:(devx-269-multi-reporters) saucectl run
➜ sauce-cypress-runner git:(devx-269-multi-reporters)
^ Running saucectl on the above config shows no output
Recommended solution: Make a validator in Go that checks the YML object before processing it and then gives helpful error messages if it wasn't parsed properly.
If you provide a string when it should be an array it gives a GoLang error like this failed to parse job configuration: yaml: unmarshal errors:\n line 9: cannot unmarshal. This may be confusing to users, so I think we need to add a YML validation step that gives human readable errors when the user provides the wrong format so it says something like found a string in the "files:" attribute when we expect an array of strings.
Ideally we should accept strings and string[] and just treat strings as a singleton array.
git clone [email protected]:dpgraham/testbed.git && cd testbed
npm install
saucectl run --config .sauce/string-files-config.yml
# Output
5:03PM ERR failed to execute run command error="failed to parse job configuration: yaml: unmarshal errors:\n line 9: cannot unmarshal !!str `cypress/` into []string"
Tunnel configuration in a yml looks like it's being ignored when using the hardware device selector (RDC mode?). Note that I'm trying to use SauceLabs tunnels, not old-style TO tunnels.
Tunnel configuration doesn't get ignored.
Write a yml that uses a SauceLabs tunnel and an Android hardware device. Try running. Tunnel will be shown as ignored.
saucectl 0.44.0
Can i run single tests on sauce just like we run using jest jest filename -t 'sampletest'
I am trying to modify the config.yml to achieve this, but couldn't get it.
apiVersion: v1
metadata:
name: Playwright and Jest Sauce Labs
tags:
- e2e testing
build: Release $CI_COMMIT_SHORT_SHA
files:
- ./tests/
test
- filename 'sampletest'
image:
base: saucelabs/stt-playwright-jest-node
version: v0.1.3
sauce:
region: us-west-1
Could you guys let me know if I am missing something here?
When I run saucectl for the first time on my laptop it complains about untrusted developer
Download saucectl v0.3.1 and run it for the first time
OS
sw_vers
ProductName: Mac OS X
ProductVersion: 10.15.4
BuildVersion: 19E266
Codesign:
codesign -vv --deep-verify ./saucectl
./saucectl: code object is not signed at all
In architecture: x86_64
As a developer I want to have a seamless integration into VSCode (and eventually other) IDE(s). Let's ensure that they can set breakpoints and start a saucectl test from the IDE directly.
Workflow:
I am trying to use POM in testcafe framework and when I try to run using saucectl I see transpilation issues.
$ sudo saucectl run
$ saucectl -v
saucectl version 0.10.0
(build 6bb5887)
Laptop ubuntu 18.0.1
While running using command testcafe chrome login.test.ts it shows no error and runs sucessfully
$ sudo saucectl run
@saucelabs/[email protected] test /home/seluser
node ./src/testcafe-runner.js
Cannot prepare tests due to an error.
Error: TypeScript compilation failed.
/home/seluser/tests/login.test.ts (2, 23): Cannot find module '../page-model/Login' or its corresponding type declarations.
at Function._reportErrors (/home/seluser/node_modules/testcafe/src/compiler/test-file/formats/typescript/compiler.ts:79:15)
at TypeScriptTestFileCompiler._compileFilesToCache (/home/seluser/node_modules/testcafe/src/compiler/test-file/formats/typescript/compiler.ts:101:40)
at TypeScriptTestFileCompiler._precompileCode (/home/seluser/node_modules/testcafe/src/compiler/test-file/formats/typescript/compiler.ts:129:18)
at TypeScriptTestFileCompiler._compileCodeForTestFiles (/home/seluser/node_modules/testcafe/src/compiler/test-file/api-based.js:111:29)
at /home/seluser/node_modules/testcafe/src/compiler/test-file/formats/typescript/compiler.ts:85:30
at Compiler._precompileFiles (/home/seluser/node_modules/testcafe/src/compiler/index.js:58:33)
at Compiler._compileTestFiles (/home/seluser/node_modules/testcafe/src/compiler/index.js:93:9)
at Compiler.getTests (/home/seluser/node_modules/testcafe/src/compiler/index.js:111:34)
at Bootstrapper._getTests (/home/seluser/node_modules/testcafe/src/runner/bootstrapper.ts:241:21)
at Bootstrapper._bootstrapParallel (/home/seluser/node_modules/testcafe/src/runner/bootstrapper.ts:391:38)
npm ERR! Test failed. See above for more details.
POM class
class Login {
mobilenumber: Selector = Selector('input').withAttribute("aria-label","mobile-number");
continue: Selector = Selector('button').withText('CONTINUE');
}
script file
import LoginPage from "../page-model/Login";
const url = 'https://xxx.com/login';
fixture('Login tests').page(url);
test('Able to move to otp page',async t => {
await t
.expect(LoginPage.mobilenumber.exists).ok();
})
Setting a timeout doesn't kill a running and considered-as-hung test session.
Enforcing --timeout with a hardkill instead of sending a CTRL+C to console
Run a test session with a while(true) loop and a short enough global timeout
0.48.1
Cypress custom commands are not recognized because it doesn't read cypress.json and therefore, the tests fail on a custom command. It does not consider it as a function.
We are using a custom command cy.login() in the beforeEach hook and this is defined in commands.js imported into index.js which is defined as a pluginFile in cypress.json. The problem is that this file is not read, omitting all the other information needed for the tests. There is no way to pass any information that we would put in cypress.json aside from the commands. Therefore, when it fails in the before each hook, it skips the remaining tests in the current suite. It doesn't generate the screenshot, but the video and job details are available on SauceLabs Automated Test Results.
Let's say the user has the following config:
apiVersion: v1
metadata:
name: Feature XYZ
tags:
- e2e
- release team
- other tag
build: Release $CI_COMMIT_SHORT_SHA
files:
- tests/exampel.test.js
image:
base: saucelabs/stt-puppeteer-node
version: latest
sauce:
region: us-west-1Note the little typo error in the files: tests/exampel.test.js which causes the following error to be thrown:
3:23PM ERR Failed to tear down environment error="Error response from daemon: page not found"
3:23PM ERR failed to execute run command error="Error response from daemon: invalid mount config for type \"bind\": bind source path does not exist: /path/to/tests/exampel.test.js"
I suggest to check for file existence before binding it to the container and provide a better error message for it.
It seems the file resolution defined in the ./sauce/config.yml "files" only works with files that are entirely self contained. In other words tests that import data or page objects from another module or use custom Cypress commands defined outside the test file etc fail.
saucectl version
saucectl version 0.11.0
(build 7608263)
Last saucectl version that did not exhibit the issue (if applicable):
unknown
Desktop OS/version used to run saucectl:
Macbook pro MacOS Mojave 10.14.3 (18D42)
Output from CLI
Oops...we found an error preparing this test file:
/home/seluser/cypress/integration/tests/budget_slider.spec.js
The error was:
Error: Cannot find module './../pages/ObjectiveCampaignCreation' from '/home/seluser/cypress/integration/tests'
This occurred while Cypress was compiling and bundling your test code. This is usually caused by:
- A missing file or dependency
- A syntax error in the file or one of its dependencies
upload failed: TypeError: Cannot read property 'startsWith' of null
at SauceLabs._uploadJobAssets (/home/seluser/node_modules/saucelabs/build/index.js:296:64)
at exports.sauceReporter (/home/seluser/src/sauce-reporter.js:94:9)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async report (/home/seluser/src/cypress-runner.js:26:5)
at async cypressRunner (/home/seluser/src/cypress-runner.js:52:20)
at async /home/seluser/bin/cypress:4:3
I can't include logs from saucelabs since the test doesn't even start
Please remember that with sample code it's easier to reproduce the bug and it's much faster to fix it.
While I am running sample test on stt-playwright-jest-node facing issue no video is result even though it shows test as passed
Issue in terminal:
[email protected] test /home/seluser
DISPLAY="$(cat DISPLAY)" DEBUG="playwright:*" jest --config=./.config/jest.config.js --runInBand
In Saucelab UI :-
Error:
Misconfigured -- Unsupported OS/browser/version/device combo: OS: '', Browser: 'googlechrome', Version: '.', Device: 'unspecified'
Also video file is blank.
test code from playwright-jest-runner
From version 0.28 of saucectl there is a functionality that will force file transfer mode to use 'copy' when concurrency > 1 This functionality does not work with docker on windows (might or might not work on other operating systems have none to try). Installing saucectl 0.27 fixes the issue since it mounts instead of copies even for concurrency > 1. If I set concurrency to 1 it works fine in later versions of saucectl also since it mounts the filesystem instead of copying. This is how it looks using the latest (as of writing) saucectl, 0.32:
This is how it looks when using ccy 1
Either the copy should be fixed so that it works on windows, or it should be possible to mount instead of copying when using concurrency mode (fileTransfer: mount in config.yml does not help)
apiVersion: v1alpha
kind: cypress
sauce:
region: eu-central-1
metadata:
name: Testing Cypress Support
tags:
- e2e
- release team
- other tag
build: "Cypress: parallel - $DATE"
docker:
fileTransfer: mount
cypress:
version: 6.6.0
configFile: "cypress.json" # We determine related files based on the location of the config file.
suites:
- name: "Cypress Chrome"
browser: "chrome"
platformName: "Windows 10"
screenResolution: "1400x1050"
config:
testFiles: [ "**/*.*" ]
saucectl: 0.32 (same problem with all versions after 0.27)
docker-image: saucelabs/stt-cypress-mocha-node:v6.0.1
operating system: Windows 10 (20H2)
docker-for-windows: 3.2.1 (docker engine 21.10.5) (tried with both wsl2 and hyper-v backend, same problem on both)
In my pipeline saucectl reports some:
2:47PM ERR failed to execute run command error="failed to locate runner configuration: open /home/seluser/config.yaml: no such file or direct
Regardless of the error the pipeline step still passes. I expect saucectl to always fail if the test could not be run.
In our case we do not follow cypress default folder structure.
our looks
when we execute 'saucectl run'
I see below error.
Running version 0.34.1
Reading config file .sauce/config.yml
failed to execute run command "unable to locate the cypress folder in ."
Is there a way I can set the path for cypress tests ?
When tests pass on retry, sauce labs make the test as failed.
Example: With cypress retry https://docs.cypress.io/guides/guides/test-retries#Introduction
Attempt 1: Test failed.
Attempt 2: Test failed.
Attempt 3: Test passed
Test is marked as failed.
When tests pass on retry, sauce labs should mark the test as passed. (at least a way to configure if not by default)
Example: With cypress retry https://docs.cypress.io/guides/guides/test-retries#Introduction
Attempt 1: Test failed.
Attempt 2: Test failed.
Attempt 3: Test passed
Test marked as passed.
Espresso supports specifying multiple packages to run tests from, and even the Legacy RDC runner supports that as well.
Currently to run tests from multiple packages I would need to create a separate tests suite for each package. That would slow down running tests if I were to run then with no concurrency, because each suite needs to bootstrap the device.
- name: "PR Tests"
testOptions:
package:
- com.domain.tests.feature
- com.domain.tests.health
- com.domain.tests.onboarding
clearPackageData: true
useTestOrchestrator: true
devices:
- name: "Samsung Galaxy.*"
platformVersion: 9
SauceCtl sometimes returns status codes of 0 when the tests aren't passing
An example is here: https://github.com/saucelabs/saucectl/blob/master/internal/ci/runner.go#L76 <-- when this line gets reached, saucectl fails but returns a status code of 0
$ saucectl --version
saucectl version 0.10.0
(build 6bb58873784a982d8d8573ee168e7d2c4553a9e4)
I have a custom test env setup (setup.jest.env.js) in my jest.config.js
module.exports = {
testTimeout: 60000,
preset: "jest-playwright-preset",
testPathIgnorePatterns: ["/node_modules/"],
testMatch: ["**/tests/spec/*.test.js"],
testEnvironment: "./setup.jest.env.js",
};
//contents of setup.jest.env.js
const PlaywrightEnvironment = require('jest-playwright-preset/lib/PlaywrightEnvironment')
.default
class CustomEnvironment extends PlaywrightEnvironment {
async setup() {
await super.setup()
console.log("setup started") //this doesn't print while saucectl run, works fine on local jest command
}
async teardown() {
// Your teardown
await super.teardown()
}
async handleTestEvent(event) {
if (event.name === 'test_done' && event.test.errors.length > 0) {
const parentName = event.test.parent.name.replace(/\W/g, '-')
const specName = event.test.name.replace(/\W/g, '-')
await this.global.page.screenshot({
path: `screenshots/${parentName}_${specName}.png`,
})
}
}
}
module.exports = CustomEnvironment
Does saucectl support custom evironments?
Can we run playwright tests by opening a session on saucelabs, just like how we do it using selenium without using saucectl?
I have some tests up and running using the puppeteer image, I noticed when the github action runs, pulling the images takes over 1 minutes, and the test itself takes 5- 10 seconds. It would be useful to be able to cache the image in the workflow container. This would speed up tests by 1 minute and tests would take seconds and not minutes.
saucectl uses v1.1 tunnels REST API endpoint /available_tunnels?full=1 to get all the tunnels for the user and their org.
The endpoint is a part of API version that Sauce Labs is planning to deprecate in the future.
saucectl should use v1 tunnels REST API endpoint /tunnels?all=true&full=true to get all the tunnels for the user and their org.
The endpoint is a part of the supported API version.
When running Cypress tests on saucelabs using saucectl, there is currently a difference between passing environment variables to --test-env sauce and --test-env docker.
When you run --test-env docker --env CYPRESS_SMOKE_USER=... the varaiable is available inside Cypress. Changing the --test-env to sauce, the variable is not passed anymore. If you want to achieve, you need to remove the CYPRESS_ from the environment variable.
Either align the variable usage or make it clearer in the documentation that there is a difference between the 2 ways of working. Afterwards it was easy, but the cypress tests were initially not running inside saucelabs but purely as a Cypress custom docker build. However when we changed them we tried different ways of passing environment variables, and it seems the docs are somewhat confusing or missing info. So updating the information with the info that you might need to remove the CYPRESS_ prefix in certain circumstances will already help future users to also look into that area.
Saucectl 0.37.0
Cypress 5.6.0
We're trying to pass environment variables using the config.yml file, and also tried using the --env flag on saucectl directly.
We're printing out the env vars via process.env and process.env.REACT_APP_AWS_PROJECT_REGION in one of the test files via console.log, and are not seeing any of the env variables we set being shown.
The env should print out with the envs passed through.
suites:
- name: "saucy test"
browserName: "chrome"
src:
- "e2e/*.test.ts" # test files glob
screenshots:
takeOnFails: true
fullPage: true
platformName: "macOS 11.00"
env:
REACT_APP_AWS_PROJECT_REGION: us-east-1
speed: 1
docker:
fileTransfer: mount
testcafe:
version: 1.14.0
rootDir: .
npm:
registry: https://registry.npmjs.org
packages:
"amazon-cognito-identity-js": "^4.3.3"
"aws-sdk": "^2.854.0"
"cross-fetch": "^3.0.5"
"faker": "^4.1.0"
"uuid": "^8.3.0"
"testcafe-react-selectors": "4.1.4"
"testcafe": "^1.14.0"
When specifying a useTestOrchestrator and clearPackageData for a test suite, the android test orchestrator is not used and the app data isn't cleared after every test.
The test orchestrator apk should be installed and the test runner should start and use the test orchestrator service
apiVersion: v1alpha
kind: espresso
sauce:
region: eu-central-1
concurrency: 1
metadata:
name: Espresso Tests
tags:
- e2e
- android
build: "PR: $BITRISE_GIT_MESSAGE"
espresso:
app: ./app/build/outputs/apk/kaufda/debug/kaufdaDebug-21.10.0.D.apk
testApp: ./app/build/outputs/apk/androidTest/kaufda/debug/app-kaufda-debug-androidTest.apk
suites:
- name: "Feature Tests"
testOptions:
package: com.bonial.tests.feature
clearPackageData: true
useTestOrchestrator: true
devices:
- name: "Samsung Galaxy.*"
platformVersion: 9
artifacts:
download:
when: always
match:
- junit.xml
directory: ./artifacts/
saucectl version 0.60.0
(build 69690284b2e4c27b44d7993e2a8a78e76a427368)
saucectl throws panic error If the docker image to use for tests is missing from the system.
panic: repository name must be canonical
goroutine 1 [running]:
github.com/saucelabs/saucectl/cli/command/run.checkErr(...)
/sauce/cli/command/run/cmd.go:47
github.com/saucelabs/saucectl/cli/command/run.NewRunCommand.func1(0xc0000b2dc0, 0xc000205140, 0x0, 0x2)
/sauce/cli/command/run/cmd.go:34 +0xe2
github.com/spf13/cobra.(*Command).execute(0xc0000b2dc0, 0xc000098060, 0x2, 0x2, 0xc0000b2dc0, 0xc000098060)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:846 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xc0000b2b00, 0xc000094008, 0xc0001c1f20, 0x1)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
main.main()
/sauce/cmd/saucectl/saucectl.go:36 +0x1ff
make: *** [functional_test_puppeteer] Error 2
Using this image: saucelabs/sauce-puppeteer-runner example test a config.yml,
remove the image with
docker rmi $(docker image ls | grep saucelabs/sauce-puppeteer-runner | awk '{print $3}')
run saucectl
OS
sw_vers
ProductName: Mac OS X
ProductVersion: 10.15.4
BuildVersion: 19E266
When running XCUITests the results do not include XCUI Runner Logs. The results include screenshots, device logs, junit.xml but miss the most important file (XCUI Runner logs). Can those be added? They are needed for debugging any test failures.
Refer to the code here -
saucectl/internal/rdc/client.go
Line 180 in 8037ba9
I have import "cypress-jest-adapter" and other deps that I use in cypress tests. But package.json that in the image doesn't contains them, so I have an error:
Error: Cannot find module 'cypress-jest-adapter' from '/home/seluser/cypress/integration/tests/integration'
saucectl version 0.10.0
(build 6bb5887)
linux 64
cypress 4.11.0
This log should be enough, 'cause the issue is pretty straightforward: Missing dependency in package.json in the image
Error: Cannot find module 'cypress-jest-adapter' from '/home/seluser/cypress/integration/tests/integration'
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.