saucelabs / testrunner-toolkit

Sauce Labs Test Runner Toolkit is a JavaScript native approach to perform browser testing in a CI with Sauce Labs

Sauce Labs Testrunner Toolkit !

Sauce Labs Testrunner Toolkit is a containerized testing solution that simplifies user setup, speeds up test execution time and supports native Javascript frameworks like Puppeteer and Playwright for running end-to-end web tests with Sauce Labs.

Getting started

To install and use Sauce Testrunner Toolkit, read more here.

Collaboration

There is a lot we can imagine doing next. It starts with hearing from you. Submit issues and features here - everything helps!

Contribution

The Sauce Labs Testrunner Toolkit is part of our commitment to a world of digital confidence where each of our digital lives and experiences are magical. If you are thinking about getting involved, please do. This repository is focused on helping people learn how to test their user experience. More is welcome.

• Contribution guidelines
• Code of conduct

testrunner-toolkit's Issues

File structure is not properly copied

🐛 Bug Report

I have a project that uses jest-puppeteer which can be found here https://github.com/saucelabs-training/demo-js/tree/master/puppeteer

I require different files in the specs and added them in my config file, see here but it results in an error. If I include the files in the same folder as the spec files the error disappears.

To Reproduce

See my project

Expected behavior

Folder structure is copied and the required files are being used

Scripts and configuration needed to reproduce this issue (when applicable)

Check out this project https://github.com/saucelabs-training/demo-js.git, cd puppeteer and run it

Logs

➜  jest-puppeteer git:(feat/jest-puppeteer) ✗ saucectl run

> [email protected] test /home/seluser
> DISPLAY="$(cat DISPLAY)" DEBUG="puppeteer:*" jest --config=./.config/jest.config.js

FAIL tests/cart.summary.spec.js
  ● Test suite failed to run

    Cannot find module '../e2eConstants' from 'cart.summary.spec.js'

    > 1 | const {LOGIN_USERS, PAGES, PRODUCTS} = require('../e2eConstants');
        |                                        ^
      2 | const SwagOverviewPage = require('../page-objects/SwagOverviewPage');
      3 | const CheckoutPersonalInfoPage = require('../page-objects/CheckoutPersonalInfoPage');
      4 | const AppHeaderPage = require('../page-objects/AppHeaderPage');

      at Resolver.resolveModule (node_modules/jest-resolve/build/index.js:299:11)
      at Object.<anonymous> (tests/cart.summary.spec.js:1:40)

FAIL tests/swag.items.list.spec.js
  ● Test suite failed to run

    Cannot find module '../e2eConstants' from 'swag.items.list.spec.js'

    > 1 | const {LOGIN_USERS, PAGES, PRODUCTS} = require('../e2eConstants');
        |                                        ^
      2 | const SwagOverviewPage = require('../page-objects/SwagOverviewPage');
      3 | const SwagDetailsPage = require('../page-objects/SwagDetailsPage');
      4 | const AppHeaderPage = require('../page-objects/AppHeaderPage');

      at Resolver.resolveModule (node_modules/jest-resolve/build/index.js:299:11)
      at Object.<anonymous> (tests/swag.items.list.spec.js:1:40)

FAIL tests/swag.item.details.spec.js
  ● Test suite failed to run

    Cannot find module '../e2eConstants' from 'swag.item.details.spec.js'

    > 1 | const {LOGIN_USERS, PAGES, PRODUCTS} = require('../e2eConstants');
        |                                        ^
      2 | const SwagOverviewPage = require('../page-objects/SwagOverviewPage');
      3 | const SwagDetailsPage = require('../page-objects/SwagDetailsPage');
      4 | const AppHeaderPage = require('../page-objects/AppHeaderPage');

      at Resolver.resolveModule (node_modules/jest-resolve/build/index.js:299:11)
      at Object.<anonymous> (tests/swag.item.details.spec.js:1:40)

FAIL tests/checkout.personal.info.spec.js
  ● Test suite failed to run

    Cannot find module '../e2eConstants' from 'checkout.personal.info.spec.js'

    > 1 | const {LOGIN_USERS, PAGES, PERSONAL_INFO} = require('../e2eConstants');
        |                                             ^
      2 | const CheckoutPersonalInfoPage = require('../page-objects/CheckoutPersonalInfoPage');
      3 | const CheckoutSummaryPage = require('../page-objects/CheckoutSummaryPage');
      4 | const CartSummaryPage = require('../page-objects/CartSummaryPage');

      at Resolver.resolveModule (node_modules/jest-resolve/build/index.js:299:11)
      at Object.<anonymous> (tests/checkout.personal.info.spec.js:1:45)

FAIL tests/menu.spec.js
  ● Test suite failed to run

    Cannot find module '../e2eConstants' from 'menu.spec.js'

    > 1 | const {LOGIN_USERS, PAGES, PRODUCTS} = require('../e2eConstants');
        |                                        ^
      2 | const MenuPage = require('../page-objects/MenuPage');
      3 | const SwagOverviewPage = require('../page-objects/SwagOverviewPage');
      4 | const CartSummaryPage = require('../page-objects/CartSummaryPage');

      at Resolver.resolveModule (node_modules/jest-resolve/build/index.js:299:11)
      at Object.<anonymous> (tests/menu.spec.js:1:40)

FAIL tests/checkout.summary.spec.js
  ● Test suite failed to run

    Cannot find module '../e2eConstants' from 'checkout.summary.spec.js'

    > 1 | const {LOGIN_USERS, PAGES, PRODUCTS} = require('../e2eConstants');
        |                                        ^
      2 | const SwagOverviewPage = require('../page-objects/SwagOverviewPage');
      3 | const CheckoutCompletePage = require('../page-objects/CheckoutCompletePage');
      4 | const CheckoutSummaryPage = require('../page-objects/CheckoutSummaryPage');

      at Resolver.resolveModule (node_modules/jest-resolve/build/index.js:299:11)
      at Object.<anonymous> (tests/checkout.summary.spec.js:1:40)

FAIL tests/checkout.complete.spec.js
  ● Test suite failed to run

    Cannot find module '../e2eConstants' from 'checkout.complete.spec.js'

    > 1 | const {LOGIN_USERS, PAGES} = require('../e2eConstants');
        |                              ^
      2 | const CheckoutCompletePage = require('../page-objects/CheckoutCompletePage');
      3 | const {setTestContext} = require('../helpers');
      4 | 

      at Resolver.resolveModule (node_modules/jest-resolve/build/index.js:299:11)
      at Object.<anonymous> (tests/checkout.complete.spec.js:1:30)

FAIL tests/login.spec.js
  ● Test suite failed to run

    Cannot find module '../e2eConstants' from 'login.spec.js'

    > 1 | const {LOGIN_USERS} = require('../e2eConstants');
        |                       ^
      2 | const LoginPage = require('../page-objects/LoginPage');
      3 | const SwagOverviewPage = require('../page-objects/SwagOverviewPage');
      4 | 

      at Resolver.resolveModule (node_modules/jest-resolve/build/index.js:299:11)
      at Object.<anonymous> (tests/login.spec.js:1:23)

Test Suites: 8 failed, 8 total
Tests:       0 total
Snapshots:   0 total
Time:        2.303s
Ran all test suites.
internal/streams/legacy.js:61
      throw er; // Unhandled stream error in pipe.
      ^

[Error: ENOENT: no such file or directory, open '/home/seluser/log.json'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/seluser/log.json'
}
npm ERR! Test failed.  See above for more details.

Environment

OS: OSX
Docker version: 2.3.0.2
SauceCTL: 0.6.1

Add Google Analytics tag

🚀 Feature Proposal

Add a Google Analytics tag for opensource.saucelabs.com

Motivation

We want to be able to track user activity in the wiki with Google Analytics. Please add the relevant gaTrackingId in siteConfig.js

Example

const siteConfig = {
     gaTrackingId: '<tracking-id>',
...
}

Build Name can't be set dynamically

🐛 Bug Report

If i set the build name in the .yml file with environment variables, these variables are always empty, even if i set them in advance. (example: https://github.com/saucelabs/testrunner-toolkit/blob/master/.sauce/puppeteer.yml#L12)

To Reproduce

Steps to reproduce the behavior (including the executed commands):

• export BUILD_ID="1234"
• export BUILD_ENV="5678"
• saucectl run

Expected behavior

Build name in sauce is "Build #1234 in 5678"

Actual behavior

Build name in sauce is "Build # in "

Environment

OS: tested on Windows 10 and Linux
Docker version: Docker version 19.03.8, build afacb8b
Saucectl version: saucectl version 0.6.3 (build ada174fe455daa5836de7586f8fecb086fc23046)
image: base: saucelabs/sauce-puppeteer version: 3.0.4

Saucelabs test runner + cypress, unable to find configuration file when running on Jenkins pipeline

🐛 Bug Report

saucectl run command fails when running on pipeline due to being unable to find the configuration file. The error is
`failed to execute run command error=failed to locate runner configuration in home/seluser/config.yaml' even though the file exists in the repository

To Reproduce

Setup a cypress project and install saucectl along with it
Setup the saucectl config file to run your cypress tests
Setup a pipeline and try to run the saucectl run command in one of the stages

Expected behavior

Saucectl run command executes and runs the cypress tests in the saucelabs environment

Scripts and configuration needed to reproduce this issue (when applicable)

Use the basic saucelabs test runner setup for cypress. Try to run the example test file in a jenkins pipeline

Environment

OS:
Docker version:

Cypress Testrunner Pass Job on GitLab Pipeline

🐛 Bug Report

Working with Cypress Toolkit on GitLab, the Pipeline is completed as a Job succeeded when there are some Tests with failures.

1. There are some Tests with failures:
   

2. The Job is passed
   

Cucumber+testcafe+sauce

Would like to design testcafe in a cucumber bdd framework and there are quite a lot of references in the internet on how to do it. But not sure how to do it with saucelabs.

Can you please guide if this is possible today with saucectl?

Is it possible to pass Cypress run options through the testrunner toolkit?

Is it possible to pass Cypress run options through the testrunner toolkit like “baseUrl”?

When I run my tests I am using the cypress command like:
cypress run --headless --browser chrome --config video=false,baseUrl=https://${CI_COMMIT_REF_SLUG} --spec "cypress/integration/build_validation/*"

But when run saucectl, it is taking the baseUrl value defined into “cypress.json” and I would like to pass a dynamic value for the baseUrl

TestRunner does not install properly on Windows 10 Docker 19.03.13

🐛 Bug Report

A clear and concise description of what the bug is.

To Reproduce

Steps to reproduce the behavior (including the executed commands):

1. Follow the instructions on the page: https://opensource.saucelabs.com/testrunner-toolkit/docs/installation
2. Insert username and accesskey as System Variables in Windows 10
3. Run command: saucectl new

Expected behavior

A selection that allows me to choose the desired framework following the prompt.

Scripts and configuration needed to reproduce this issue (when applicable)

Follow the instructions on page: https://opensource.saucelabs.com/testrunner-toolkit/docs/installation

Docker info:

Docker version 19.03.13, build 4484c46d9d

C:\Users\<username>>docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.128-microsoft-standard
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 6.136GiB
 Name: docker-desktop
 ID: 6VMS:D27P:EABL:YTVL:JLGD:Q5PG:XDPU:AMMQ:CGCG:ARSF:LNMI:IQZH
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Error returned when running saucectl -v

Error:
C:\Users\<username>>saucectl -v
(node:17348) UnhandledPromiseRejectionWarning: HTTPError: Response code 404 (Not Found)
    at EventEmitter.<anonymous> (C:\Users\<username>\AppData\Roaming\npm\node_modules\saucectl\node_modules\got\index.js:482:24)
    at EventEmitter.emit (events.js:315:20)
    at getResponse (C:\Users\<username>\AppData\Roaming\npm\node_modules\saucectl\node_modules\got\index.js:320:5)
    at Immediate.<anonymous> (C:\Users\<username>\AppData\Roaming\npm\node_modules\saucectl\node_modules\got\index.js:147:6)
    at processImmediate (internal/timers.js:456:21)
(node:17348) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:17348) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Environment

OS: Windows 10
Docker version: 19.03.13

Use jest-puppeteer or add `page` as a global

🚀 Feature Proposal

When I started with Puppeteer and Jest I checked the internet and found https://github.com/smooth-code/jest-puppeteer. This is a project that already doest everything for you so you only need to write your tests. The advantage of this project is that it's well maintained and used by a lot of people (around 100-125K downloads a week https://npm-stat.com/charts.html?package=jest-puppeteer).
The beauty if this project is that is also provides page as a global making the code cleaner (not providing page each time to methods that need to use page.

The SauceCTL has it's own implementation of Jest and Puppeteer, this might have advantages, but also limits people who want to use jest-puppeteer, already have a set of jest-puppeteer tests and want to use SauceCTL to store the assets in Sauce. This is currently not working out of the box. It would be nice if we could:

• or use the lib instead of our own implementation, it has some nice features so we can extend it by for example creating our own setup or tear down, see https://github.com/smooth-code/jest-puppeteer#create-your-own-globalsetup-and-globalteardown
• or expose the page as a global, so tests written with this project can be re-used without having to rewrite them / edit tests to run them on Sauce, see also https://github.com/smooth-code/jest-puppeteer#api

This will also be more in line with the Playwright setup, which also has the page as a global

Example

https://github.com/saucelabs-training/demo-js/tree/master/puppeteer

Get rid of the `await` and get us that nice wdio API

🚀 Feature Proposal

That webdriver IO API is super nice without having to write await for every single line of code. Let's get that integrated here please

Motivation

Webdriver IO, duplication

Example

const page = (await browser.pages())[0]
page.goto('https://www.saucedemo.com/');
page.type('#user-name', 'standard_user');

Video is not available

🐛 Bug Report

Video is not available after running tests (error while running)
Example on Sauce: https://app.saucelabs.com/tests/b0b16667a0e64d85bf8720139202f214

$ saucectl run

> [email protected] test /home/seluser
> DISPLAY="$(cat DISPLAY)" DEBUG="puppeteer:*" jest --config=./.config/jest.config.js --runInBand

cat: DISPLAY: No such file or directory
Watch test: https://chrome-devtools-frontend.appspot.com/serve_file/@ec99b9f060de3f065f466ccd2b2bfbf17376b61e/devtools_app.html?ws=localhost:9222/devtools/page/3FA97E9D36927AF8AFA340BE5EFF9C0F&remoteFrontend=true

PASS tests/example.test.js
  saucectl demo test
    ✓ should verify title of the page (576ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        3.645s
Ran all test suites.

Open job details page: https://app.saucelabs.com/tests/b0b16667a0e64d85bf8720139202f214

To Reproduce

Steps to reproduce the behavior (including the executed commands):
npm i -g saucectl
set Sauce Labs environment variables in terminal
saucectl new (select puppeteer)
saucectl run

Expected behavior

A clear and concise description of what you expected to happen.

Scripts and configuration needed to reproduce this issue (when applicable)

N/A

Environment

OS: Windows 10
Docker version: Docker version 19.03.8, build afacb8b

we should have the possibility to define the sauce labs data center

🚀 Feature Proposal

there should be a possibility to define the sauce labs data center to upload test results to the eu data center

Issue while running on Windows with Git Bash

🐛 Bug Report

USER@COMPUTER MINGW64 /d/Repos/saucectl (master)
$ saucectl run
panic: open D:\Repos\saucectl/C:\Users\USER\AppData\Local\Temp\saucectl592405251\config.yaml: The filename, directory name, or volume label syntax is incorrect.

goroutine 1 [running]:
github.com/saucelabs/saucectl/cli/command/run.checkErr(...)
        /home/runner/work/saucectl-internal/saucectl-internal/cli/command/run/cmd.go:48
github.com/saucelabs/saucectl/cli/command/run.Command.func1(0x11cbac60, 0xc297c4, 0x0, 0x0)
        /home/runner/work/saucectl-internal/saucectl-internal/cli/command/run/cmd.go:35 +0xbb
github.com/spf13/cobra.(*Command).execute(0x11cbac60, 0x11c940e8, 0x0, 0x0, 0x11cbac60, 0x11c940e8)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:846 +0x1e3
github.com/spf13/cobra.(*Command).ExecuteC(0x11cba9a0, 0x11c95b40, 0x7, 0x0)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x291
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
main.main()
        /home/runner/work/saucectl-internal/saucectl-internal/cmd/saucectl/saucectl.go:48 +0x2f7

To Reproduce

1. npm i -g saucectl
2. set Sauce Labs environment variables in terminal
3. saucectl new (select puppeteer)
4. saucectl run

Expected behavior

I expected the test to run without errors

Scripts and configuration needed to reproduce this issue (when applicable)

N/A

Environment

OS: Windows 10
Docker version: Docker version 19.03.8, build afacb8b

running with git bash on windows

Curl installation failed for updating the version

🐛 Bug Report

I already have SauceCTL 0.4.1 installed on my machine, if I want to upgrade to the latest version I use this

curl -L https://saucelabs.github.io/saucectl/install | bash

I got this error

➜  js-sauce-integrations git:(feat/jest-puppeteer) ✗ curl -L https://saucelabs.github.io/saucectl/install | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9664  100  9664    0     0  91169      0 --:--:-- --:--:-- --:--:-- 91169
saucelabs/saucectl INFO checking GitHub for latest tag
saucelabs/saucectl INFO found version: 0.6.1 for v0.6.1/mac/64-bit
saucelabs/saucectl DEBUG downloading tarball https://github.com/saucelabs/saucectl/releases/download/v0.6.1/saucectl_0.6.1_mac_64-bit.tar.gz
saucelabs/saucectl DEBUG downloading checksum https://github.com/saucelabs/saucectl/releases/download/v0.6.1/checksums.txt
saucelabs/saucectl DEBUG verify tarball
saucelabs/saucectl DEBUG Preparing to install saucectl
mv: /var/folders/vg/lfm83td92t921fbqjrxrs4ym0000gp/T/saucectl-internal: No such file or directory

and the version is not upgraded

To Reproduce

See above

Expected behavior

It would upgrade to the latest version

Scripts and configuration needed to reproduce this issue (when applicable)

See above

Environment

OS: OSX
Docker version: 2.3.0.2

Throw proper error if install script is not able to fetch GitHub API

🚀 Feature Proposal

It would be nice if the install script could check each requests response code to properly error out when the API can't be access (e.g. due to rate limiting or network issues).

Motivation

Attempts to install saucectl fail with this error message:

curl -L https://saucelabs.github.io/saucectl/install | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9664  100  9664    0     0  20008      0 --:--:-- --:--:-- --:--:-- 19966
saucelabs/saucectl INFO checking GitHub for latest tag
saucelabs/saucectl INFO found version: { for {/mac/64-bit
saucelabs/saucectl DEBUG downloading tarball https://github.com/saucelabs/saucectl/releases/download/{/saucectl_{_mac_64-bit.tar.gz
curl: (3) nested brace in URL position 68:
https://github.com/saucelabs/saucectl/releases/download/{/saucectl_{_mac_64-bit.tar.gz

all tests are reported in one sauce labs session

🐛 Bug Report

all tests are reported in one sauce labs session instead of a separate session per file

To Reproduce

• execute test with multiple test files

Expected behavior

each test file should be reported in a separate sauce session.
-> separate video for each test file

Environment

OS: tested on Windows 10 and Linux
Docker version: Docker version 19.03.8, build afacb8b
Saucectl version: saucectl version 0.6.3 (build ada174fe455daa5836de7586f8fecb086fc23046)
image: base: saucelabs/sauce-puppeteer version: 3.0.4

Environment variable support

🚀 Feature Proposal

We need a way to define some environment variables. if i access the environment variables with "process.env.VARIABLE" i get "undefined". Probably due to the fact that these environment variables aren't available in the docker container.
Can we somehow pass the environment variable to the docker container with the saucectl command?

Tests aren't executing my commands

🐛 Bug Report

I am expecting one of my tests to type stuff into text fields, but that's not happening.

To Reproduce

1. Get my code from this branch
2. Navigate to saucectl/examples directory
3. Do saucectl run
4. Get output like this https://app.saucelabs.com/tests/d4f7fb331b3a47c8aa4eb2c758aac1df#89

Here are the sample tests just in case:

describe('SauceDemo login page', () => {
	test('should load', async () => {
		const page = (await browser.pages())[0]
		await page.goto('https://www.saucedemo.com/');
		expect(await page.title()).toBe('Swag Labs');
	});
	test('should login', async () => {
		const page = (await browser.pages())[0]
		await page.goto('https://www.saucedemo.com/');
		await page.type('#user-name', 'standard_user');
		await page.type('#password', 'secret_sauce');
		await page.click('.btn_action');
		expect(await page.title()).toBe('Swag Labs');
	});
});

Expected behavior

Typing text should work

Scripts and configuration needed to reproduce this issue (when applicable)

Please provide the scripts and configuration to reproduce the issue you
are reporting, if the setup is more complex, GitHub repo links are also OK.

Issues without a reproduction script are likely to stall and eventually be closed.

Environment

OS: macOS Catalina
Docker version: 2.3.0.3

🍵 TestCafe?

🚀 Feature Proposal

Does saucectl have plans to support TestCafe?

• https://devexpress.github.io/testcafe/

Motivation

I have TestCafe tests I'd like to run on Sauce.

Example

https://github.com/software-platform/web_test_infra/tree/master/TestCafe

Testrunner Toolkit for Cypress stop testing after first failure

🐛 Bug Report

I am executing regression tests with Testrunner toolkit for Cypress, but at the first failure the testing stop.
How can I define to continue with the testing after get any failure during the tests execution?

Tests run with SauceCtl and testcafe much longer than with local execution

🐛 Bug Report

I see a big time difference between local (original Testcafe) and the STT solution. It's 90+ seconds versus 4+ minutes in Docker
Secondly I see that in the STT solution 75 tests are executed versus the original 25 (they contain the same tests). I also see a lot of double tests in the logs

To Reproduce

• This is the local set up for running TestCafe on your local machine, just follow the steps to get it running locally https://github.com/saucelabs-training/demo-js/tree/master/testcafe
• This is the STT setup, https://github.com/saucelabs-training/demo-js/tree/master/testrunner-toolkit/testcafe, install and run

Expected behavior

I expect that there is not a lot of time difference between local and the docker execution and the amount of tests are the same

Local Output

yarn test.local                                                                                                            
yarn run v1.22.10
$ node tests/configs/local.chrome.config.js
 Running tests in:
 - Chrome 86.0.4240.198 / macOS 10.15.7
 Test cart items overview page
 ✓ should validate that we can continue shopping
 ✓ should validate that we can go from the cart to the checkout page
 ✓ should validate that a product can be removed from the cart
 Test cart items overview page
 ✓ should validate that we can continue shopping
 Test checkout overview page
 ✓ should validate we get an error if we don not provide all personal information
 ✓ should validate that we can cancel the first checkout
 ✓ should be able to continue the checkout
 Test checkout summary page
 ✓ should validate that we can continue shopping
 ✓ should validate that we can cancel checkout and go to the inventory page
 ✓ should validate that we have 1 product in our checkout overview
 Test Login
 ✓ should be able to test loading of login page
 ✓ should be able to login with a standard user
 ✓ should not be able to login with a locked user
 Test checkout summary page
 ✓ should be able to go to the swag items overview page
 ✓ should be able to open the about page
 ✓ should be able to log out
 ✓ should be able to clear the cart
 Test swag items details page
 ✓ should validate that we can go back from the details to the inventory page
 ✓ should validate that a product can be added to a cart
 ✓ should validate that a product can be removed from the cart
 Test swag items overview page
 ✓ should validate that all products are present
 ✓ should validate that the details of a product can be opened
 ✓ should validate that a product can be added to a cart
 ✓ should validate that a product can be removed from the cart
 ✓ should be able to open the cart summary page
 25 passed (1m 26s)
Tests failed: 0
✨  Done in 98.22s.

Saucectl:

saucectl run
1:32PM INF Start Run Command
1:32PM INF Reading config file config=.sauce/config.yml
1:32PM INF Starting local runner
1:32PM INF Setting up test environment
⠋ Starting container saucelabs/stt-testcafe-node:latest 1:32PM INF File mounted from=./tests/ to=/home/seluser/tests
1:32PM INF Starting tests
> [email protected] test /home/seluser
> node ./src/testcafe-runner.js
 Running tests in:
 - Chrome 81.0.4044.138 / Linux 0.0
 ✓ Test cart items overview page - should validate that we can continue
 shopping
 ✓ Test cart items overview page - should validate that we can go from the
 cart to the checkout page
 ✓ Test cart items overview page - should validate that a product can be
 removed from the cart
 ✓ Test cart items overview page - should validate that we can continue
 shopping
 ✓ Test checkout overview page - should validate we get an error if we don
 not provide all personal information
 ✓ Test checkout overview page - should validate that we can cancel the first
 checkout
 ✓ Test checkout overview page - should be able to continue the checkout
 ✓ Test checkout summary page - should validate that we can continue shopping
 ✓ Test checkout summary page - should validate that we can cancel checkout
 and go to the inventory page
 ✓ Test checkout summary page - should validate that we have 1 product in our
 checkout overview
 ✓ Test Login - should be able to test loading of login page
 ✓ Test Login - should be able to login with a standard user
 ✓ Test Login - should not be able to login with a locked user
 ✓ Test checkout summary page - should be able to go to the swag items
 overview page
 ✓ Test checkout summary page - should be able to open the about page
 ✓ Test checkout summary page - should be able to log out
 ✓ Test checkout summary page - should be able to clear the cart
 ✓ Test swag items details page - should validate that we can go back from
 the details to the inventory page
 ✓ Test swag items details page - should validate that a product can be added
 to a cart
 ✓ Test swag items details page - should validate that a product can be
 removed from the cart
 ✓ Test swag items overview page - should validate that all products are
 present
 ✓ Test swag items overview page - should validate that the details of a
 product can be opened
 ✓ Test swag items overview page - should validate that a product can be
 added to a cart
 ✓ Test swag items overview page - should validate that a product can be
 removed from the cart
 ✓ Test swag items overview page - should be able to open the cart summary
 page
 ✓ Test cart items overview page - should validate that we can continue
 shopping
 ✓ Test cart items overview page - should validate that we can go from the
 cart to the checkout page
 ✓ Test cart items overview page - should validate that a product can be
 removed from the cart
 ✓ Test cart items overview page - should validate that we can continue
 shopping
 ✓ Test checkout overview page - should validate we get an error if we don
 not provide all personal information
 ✓ Test checkout overview page - should validate that we can cancel the first
 checkout
 ✓ Test checkout overview page - should be able to continue the checkout
 ✓ Test checkout summary page - should validate that we can continue shopping
 ✓ Test checkout summary page - should validate that we can cancel checkout
 and go to the inventory page
 ✓ Test checkout summary page - should validate that we have 1 product in our
 checkout overview
 ✓ Test Login - should be able to test loading of login page
 ✓ Test Login - should be able to login with a standard user
 ✓ Test Login - should not be able to login with a locked user
 ✓ Test checkout summary page - should be able to go to the swag items
 overview page
 ✓ Test checkout summary page - should be able to open the about page
 ✓ Test checkout summary page - should be able to log out
 ✓ Test checkout summary page - should be able to clear the cart
 ✓ Test swag items details page - should validate that we can go back from
 the details to the inventory page
 ✓ Test swag items details page - should validate that a product can be added
 to a cart
 ✓ Test swag items details page - should validate that a product can be
 removed from the cart
 ✓ Test swag items overview page - should validate that all products are
 present
 ✓ Test swag items overview page - should validate that the details of a
 product can be opened
 ✓ Test swag items overview page - should validate that a product can be
 added to a cart
 ✓ Test swag items overview page - should validate that a product can be
 removed from the cart
 ✓ Test swag items overview page - should be able to open the cart summary
 page
 ✓ Test cart items overview page - should validate that we can continue
 shopping
 ✓ Test cart items overview page - should validate that we can go from the
 cart to the checkout page
 ✓ Test cart items overview page - should validate that a product can be
 removed from the cart
 ✓ Test cart items overview page - should validate that we can continue
 shopping
 ✓ Test checkout overview page - should validate we get an error if we don
 not provide all personal information
 ✓ Test checkout overview page - should validate that we can cancel the first
 checkout
 ✓ Test checkout overview page - should be able to continue the checkout
 ✓ Test checkout summary page - should validate that we can continue shopping
 ✓ Test checkout summary page - should validate that we can cancel checkout
 and go to the inventory page
 ✓ Test checkout summary page - should validate that we have 1 product in our
 checkout overview
 ✓ Test Login - should be able to test loading of login page
 ✓ Test Login - should be able to login with a standard user
 ✓ Test Login - should not be able to login with a locked user
 ✓ Test checkout summary page - should be able to go to the swag items
 overview page
 ✓ Test checkout summary page - should be able to open the about page
 ✓ Test checkout summary page - should be able to log out
 ✓ Test checkout summary page - should be able to clear the cart
 ✓ Test swag items details page - should validate that we can go back from
 the details to the inventory page
 ✓ Test swag items details page - should validate that a product can be added
 to a cart
 ✓ Test swag items details page - should validate that a product can be
 removed from the cart
 ✓ Test swag items overview page - should validate that all products are
 present
 ✓ Test swag items overview page - should validate that the details of a
 product can be opened
 ✓ Test swag items overview page - should validate that a product can be
 added to a cart
 ✓ Test swag items overview page - should validate that a product can be
 removed from the cart
 ✓ Test swag items overview page - should be able to open the cart summary
 page
 75 passed (5m 20s)
Preparing assets
upload successful
Open job details page: https://app.eu-central-1.saucelabs.com/tests/[redacted]
1:38PM INF Command Finished ExitCode=0
1:38PM INF Tearing down environment

Assets for Cypress can't be uploaded

🐛 Bug Report

When running a cypress test I get an error when wanting to upload the assets

To Reproduce

See https://github.com/saucelabs-training/demo-js/tree/master/testrunner-toolkit/cypress

Expected behavior

Assets are uploaded

Logs

saucectl run          
8:17AM INF Start Run Command
8:17AM INF Reading config file config=.sauce/config.yml
8:17AM INF Starting local runner
8:17AM INF Setting up test environment
8:17AM INF File mounted from=./tests/ to=/home/seluser/tests
8:17AM INF Using credentials from Environment
8:17AM INF Starting tests

> [email protected] test /home/seluser
> ./bin/cypress

It looks like this is your first time using Cypress: 4.9.0

[08:17:25]  Verifying Cypress can run /home/seluser/.cache/Cypress/4.9.0/Cypress [started]
[08:17:28]  Verified Cypress!       /home/seluser/.cache/Cypress/4.9.0/Cypress [title changed]
[08:17:28]  Verified Cypress!       /home/seluser/.cache/Cypress/4.9.0/Cypress [completed]

Opening Cypress...
Couldn't determine Mocha version

tput: No value for $TERM and no -T specified
================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    4.9.0                                                                              │
  │ Browser:    Chrome 81                                                                          │
  │ Specs:      2 found (tests/cypress.spec.js, tests/cypress2.spec.js)                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  tests/cypress.spec.js                                                           (1 of 2)

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        5                                                                                │
  │ Passing:      5                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     5 seconds                                                                        │
  │ Spec Ran:     tests/cypress.spec.js                                                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  tests/cypress2.spec.js                                                          (2 of 2)

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        5                                                                                │
  │ Passing:      5                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     4 seconds                                                                        │
  │ Spec Ran:     tests/cypress2.spec.js                                                           │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


tput: No value for $TERM and no -T specified
================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  tests/cypress.spec.js                    00:05        5        5        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  tests/cypress2.spec.js                   00:04        5        5        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:09       10       10        -        -        -  

Preparing assets for tests/cypress.spec.js
upload failed: TypeError: Cannot read property 'startsWith' of null
    at SauceLabs._uploadJobAssets (/home/seluser/node_modules/saucelabs/build/index.js:296:64)
    at exports.sauceReporter (/home/seluser/src/sauce-reporter.js:94:9)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async report (/home/seluser/src/cypress-runner.js:56:5)
    at async cypressRunner (/home/seluser/src/cypress-runner.js:118:20)
    at async /home/seluser/bin/cypress:4:3

Open job details page: https://app.eu-central-1.saucelabs.com/tests/0507755cb74945308760e3942218ff0b

Preparing assets for tests/cypress2.spec.js
upload failed: TypeError: Cannot read property 'startsWith' of null
    at SauceLabs._uploadJobAssets (/home/seluser/node_modules/saucelabs/build/index.js:296:64)
    at exports.sauceReporter (/home/seluser/src/sauce-reporter.js:94:9)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async report (/home/seluser/src/cypress-runner.js:56:5)
    at async cypressRunner (/home/seluser/src/cypress-runner.js:118:20)
    at async /home/seluser/bin/cypress:4:3

Open job details page: https://app.eu-central-1.saucelabs.com/tests/885244e9864242f098ce6ed9d967a213

8:17AM INF Command Finished ExitCode=0
8:17AM INF Tearing down environment

saucectl source code?

Is the saucectl source code available? Debugging problems is difficult without source access.

Using tunnelIdentifier with testrunner toolkil?

I'm wondering if saucectl currently has any capability to set a tunnelIdentifier as you would in a sauce:config when attempting to utilize an active tunnel.

If not, does the testrunner toolkit support running tests through tunnels at all / is there plans to do so?

Apologies if this is the incorrect place to ask!