googlechromelabs / sw-testing-helpers

A set of helper files used to test Propel and sw-toolbox.

sw-testing-helpers

Testing service workers is a new space and this module is the result of some exploration in this space.

You can find docs here.

License

Copyright 2015 Google, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

sw-testing-helpers's Issues

Add Tests for Final Bundle

• Test the path of main in package.json exists.
• Test methods of class works as expected

Testing Firefox V47+

cc @addyosmani @wibblymat @jeffposnick

Just stumbled upon this bug SeleniumHQ/selenium#1395 In there one of the maintainers mentions that Firefox is working on a new WebDriver:

wolftian the plan is to use Mozilla's driver:
https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/WebDriver

But since it is not a 1.0 yet, we're likely going to not have the ability to use webdriver with Firefox versions 44 - 46(?). That is if the require addon signature does ship in 44, but even if it doesn't ship there's other breaking changes in that release for the current SeleniumHQ driver. Basically be prepared to freeze the version of firefox you have installed to 43 and then upgrade later after announcements are made that Marionette is GA with the release channel.

I'm seeing problems across all of the projects trying to test Firefox 47+ :(

We may have to skip testing on Firefox Beta until this gets ironed out, this can be done with something like:

if (browserInfo.getSeleniumBrowserId() === 'firefox' &&
  (browserInfo.getReleaseName() === 'beta' ||
  browserInfo.getReleaseName() === 'unstable')) {
  return;
}

Several scripts in projects need to quote paths

The project scripts (publish-release, etc.) are broken if your current path includes a space.

You need to put quotes around them.

e.g.

$SCRIPTPATH/publish-docs.sh releases/$PACKAGE_VERSION

should be

"$SCRIPTPATH/publish-docs.sh" releases/$PACKAGE_VERSION

activateSW should wait for iframe to be controlled by SW

Try running this:

console.log('iframe sw => ', iframe.contentWindow.navigator.serviceWorker.getRegistration().then(reg => console.log(reg)));

If shouldn't be undefined

Helpers for page navigation

In the course of writing tests for GoogleChrome/workbox#1, I found it necessary to script page navigations, running a set of tests before and after each navigation. I did this by "dropping down a level" and using the WebDriver directly to perform .get(url1), then later .get(url2), etc. This worked, but I wonder what folks think about providing a generic helper that could automate it a bit.

Add tests for Jekyll Build

Add a way to "waitForController()" instead of "activateSW()"

This is based off of feedback as a result of skipWaiting and claim in: GoogleChrome/workbox#242

Move to selenium-webdriver 3.0 betas

Unfortunately, it looks like there is development being done in the 3.0 beta releases of selenium-webdriver that's needed for compatibility with Marionette. I'm running into this, in particular: SeleniumHQ/selenium#2279

I need to call driver.manage().timeouts().setScriptTimeout(TIMEOUT) to make driver.executeAsyncScript() work properly, but that's broken when using Marionette + the selenium-webdriver 2.53.3 release.

Fix up order of releases

https://googlechrome.github.io/sw-testing-helpers/

Use node-geckodriver instead of shell script

@vladikoff just published https://github.com/vladikoff/node-geckodriver (thanks!), which allows you to fetch geckodriver as an npm dependency. We should use it in the same way we're using the chromedriver module, and do away with get-marionette.sh.

Helper for tests that require asynchronous setup in the controlled page

In the course of writing GoogleChrome/workbox#1 I've found a few situations where I need to run a test that executes code on the controlled page via WebDriverexecuteScriptAsync(), but that script needs to wait until something has happened on the controlled page before it's executed.

To give a concrete example, at https://github.com/GoogleChrome/sw-helpers/pull/1/files#diff-a18887f4e658273661a3d4f9f0f4f364R155, I'm testing whether a master entry for a page with an App Cache manifest is present in the Cache Storage API, but I need to wait until after App Cache manifest helper code on the page has run before performing that test.

I've been working around this by delaying the WebDriverexecuteScriptAsync() call via setTimeout(), but that's obviously ugly and error prone. I could imagine a runtime library provided by sw-testing-helpers creating a global readyToTest promise that the code being tested would resolve when setup was complete. That might also require the code being tested to detect when it's being run via WebDriver if we wanted to avoid that promise leaking out into production code.

npm run browser-tests leads to module not found error

I'm trying to get started with sw-testing-helpers for a new project. The README says that I can use npm run browser-tests to run the placeholder browser tests, but doing that leads to:

[~/git/sw-testing-helpers]$ npm run browser-tests

> [email protected] browser-tests /Users/jeffy/git/sw-testing-helpers
> nodemon test/server/index.js

28 Mar 11:54:54 - [nodemon] 1.7.1
28 Mar 11:54:54 - [nodemon] to restart at any time, enter `rs`
28 Mar 11:54:54 - [nodemon] watching: *.*
28 Mar 11:54:54 - [nodemon] starting `node test/server/index.js ./src/node-helpers/index.js`
module.js:341
    throw err;
    ^

Error: Cannot find module '/Users/jeffy/git/sw-testing-helpers/test/server/index.js'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Function.Module.runMain (module.js:447:10)
    at startup (node.js:142:18)
    at node.js:939:3
28 Mar 11:54:54 - [nodemon] app crashed - waiting for file changes before starting...

What commands do run to execute an initial placeholder browser test suite, that I could then modify to suit my testing needs?

Expand on the errors for automated tests

Two parts to this:

1. Make the logging a simple method that can be called from SW, Browser and Node
2. Do something along the lines:

const getFriendlyTestResult = testResult => {
  return {
    parentTitle: testResult.parent.title,
    title: testResult.title,
    state: testResult.state,
    err: testResult.err
  };
};

        if (testResults.failed.length > 0) {
          const failedTests = testResults.failed;
          let errorMessage = 'Issues in ' + loadedSW + '.\n\n' + loadedSW +
            ' had ' + testResults.failed.length + ' test failures.\n';
          errorMessage += '------------------------------------------------\n';
          errorMessage += failedTests.map((failedTest, i) => {
            return `[Failed Test ${i + 1}]\n` +
                   `    - ${failedTest.parentTitle} > ${failedTest.title}\n` +
                   `        ${failedTest.err.message}\n`;
          }).join('\n');
          errorMessage += '------------------------------------------------\n';
          throw new Error(errorMessage);
        }

Make `npm run test` happy with Firefox

The automated test suite is currently unhappy with Firefox. I've tried updating this project's dependencies, including selenium-assistant, as well as manually forcing the latest versions of both geckodriver and selenium-webdriver using npm shrinkwrap, but have not had any luck making it happy.

You can see a failed build at https://travis-ci.org/GoogleChrome/sw-testing-helpers/jobs/216046911 and the specific error is

WebDriverError: Unable to parse new session response: {"value": {"sessionId":"1e78f311-3ce2-4406-b30f-af4eabff9d51","value":{"acceptInsecureCerts":false,"browserName":"firefox","browserVersion":"52.0.2","moz:accessibilityChecks":false,"moz:processID":5260,"moz:profile":"/tmp/rust_mozprofile.G0oaFqNIjgRC","pageLoadStrategy":"normal","platformName":"linux","platformVersion":"4.8.12-040812-generic","rotatable":false,"specificationLevel":0,"timeouts":{"implicit":0,"page load":300000,"script":30000}}}}

When an npm step isn't avaiable - don't fail completely

• apart from test.

Add link to Docs on readme

https://googlechrome.github.io/sw-testing-helpers/

When npm publish fails - need to not release on github?