ahmadnassri / node-nightwatch-accessibility

Nightwatch.js utility assertion for accessibility testing with aXe

Nightwatch Accessibility

Nightwatch.js utility assertion for accessibility testing with aXe.

Install

npm install nightwatch-accessibility

Usage

Nightwatch >= 2.x

add nightwatch-accessibility to the plugins array

{
  plugins: ['nightwatch-accessibility']
}

Nightwatch <= 1.x

{
  custom_commands_path: ["./node_modules/nightwatch-accessibility/nightwatch/commands"],
  custom_assertions_path: ["./node_modules/nightwatch-accessibility/nightwatch/assertions"]
}

Use in your tests:

module.exports = {
  'Test': function (browser) {
    browser
      // initiate aXe
      .initAccessibility() 

      // execute accessibility check
      .assert.accessibility('#app', {
        verbose: true,
        rules: {
          'color-contrast': { enabled: false }
        }
      })
  }
}

API

browser.initAccessibility()

Injects the aXe library into the current test page.

browser.assert.accessibility(context, options)

Analyzes the defined context against applied aXe rules

Name	Type	Default	Description
context	String	'html'	aXe Context Parameter
options	Object	null	aXe Options Parameter

In addition to the standard options:

• options.verbose set to true will log all successful aXe tests.
• options.timeout configures the nightwatch timeout, default value is 500 milliseconds

---

Author: Ahmad Nassri • Twitter: @AhmadNassri

node-nightwatch-accessibility's Issues

Support Chrome v77 on nightwatch-accessibility

We are getting the following error on running accessibility check with the latest chrome.
Are there plan to support the Chrome v77?

thank you,
-Phil

====
Error while running .executeScriptAsync() protocol action: script timeout

✖ failed
  Failed [fail]: (aXe failed to execute)
  Failed [fail]: (aXe failed to execute)
      at process._tickCallback (internal/process/next_tick.js:68:7)

Error while running .executeScriptAsync() protocol action: script timeout

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Preset name not found within published preset config (monorepo:angularmaterial). Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

Error while injecting script

Hi,

I made a simple example to test your lib :

module.exports = {
    'Test accessibility on Home page': function (browser) {
        browser
            .url('http://www.')
            .waitForElementVisible('body', 20000)
            .initAccessibility()
            .assert.accessibility(undefined, { // Must specify default values to avoid script error
                verbose: true,
                rules: {
                    'color-contrast': { enabled: false },
                },
            })
            .end();
    },
};

I run nightwatch -c ./nightwatch.config.js and I got this error

Error while running .executeScriptAsync() protocol action: javascript error: Cannot read property 'include' of null

And here is the error happens on test :

 ✖ AccessibilityTest
 – Test accessibility on Home page (7.35s)

 Error while running .setTimeoutsAsyncScript() protocol action: Unable to find command matching POST to /timeouts/async_script

 Error while running .executeScriptAsync() protocol action: Unable to find command matching POST to /execute_async

I use:
- nightwatch: "1.1.12"
- nightwatch-accessibility: "1.8.0"

Selenium is ran inside the official docker image.

Thanks for help

error Unresolved function or method initAccessibility()

when setting up tests initAccessibility() is highlighted with error Unresolved function or method initAccessibility() and if running tests error is:"✖ Failed [fail]: (aXe failed to execute)
at processTicksAndRejections (internal/process/task_queues.js:93:5)

Error while running .setTimeoutsAsyncScript() protocol action: unknown command: unknown command: timeouts/async_script

Error while running .executeScriptAsync() protocol action: unknown command: unknown command: execute_async"
my test is:
`module.exports
= {
'@tags': ['accessibility'],

'Test Create Benefits detail page':  function(browser){
    browser
        .url(siteUrl)
        .pause(1000)
        .setValue('input[name="name"]', name)
        .setValue('input[name="pass"]', pass)
        .click('input[id="edit-submit"]')
        .url(siteUrl+'/node/add/page')
        .waitForElementVisible('.page-title', 6000)
        .assert.title('Create Benefits detail page | VA CMS')
        .pause(5000)
        .initAccessibility()
        .assert.accessibility(axeOptions, contextOptions)
        .end(function(err, res){
            console.log(res, err);
        });
},

};and nightwatch.json is {
"src_folders" : ["./tests/nightwatch/features"],
"output_folder" : "./tests/nightwatch/reports",
"custom_commands_path": ["./node_modules/nightwatch-accessibility/commands"],
"custom_assertions_path": ["./node_modules/nightwatch-accessibility/assertions"],
"page_objects_path" : "./tests/nightwatch/pageobjects",
"globals_path" : "globalsModule.js",

"selenium" : {
    "start_process" : false,
    "server_path" : "",
    "log_path" : "",
    "port" : 9515,
    "cli_args" : {
        "webdriver.chrome.driver" : "./node_modules/chromedriver/lib/chromedriver",
        "webdriver.gecko.driver" : "",
        "webdriver.edge.driver" : ""
    }
},

"test_settings" : {
    "default" : {
        "launch_url" : "http://va-gov-cms.lndo.site/",
        "selenium_port"  : 9515,
        "selenium_host"  : "localhost",
        "default_path_prefix" : "",
        "silent": true,
        "screenshots" : {
            "enabled" : false,
            "path" : ""
        }
    },
    "desiredCapabilities": {
        "browserName": "chrome",
        "chromeOptions" : {
            "args" : ["headless", "--no-sandbox"]
        },
        "acceptSslCerts": true
    },
    "chrome" : {
        "desiredCapabilities": {
            "browserName": "chrome"
        }
    }
}

}
`
"nightwatch": "^1.2.2",
"nightwatch-accessibility": "^1.8.0",
nodeJs v12.10.0

Action required: Greenkeeper could not be activated 🚨

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because we are using your CI build statuses to figure out when to notify you about breaking changes.

Since we did not receive a CI status on the greenkeeper/initial branch, we assume that you still need to configure it.

If you have already set up a CI for this repository, you might need to check your configuration. Make sure it will run on all new branches. If you don’t want it to run on every branch, you can whitelist branches starting with greenkeeper/.

We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

Once you have installed CI on this repository, you’ll need to re-trigger Greenkeeper’s initial Pull Request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper integration’s white list on Github. You'll find this list on your repo or organiszation’s settings page, under Installed GitHub Apps.

Connection refused when accessibility tests fail

When an accessibility test fails, additional errors are thrown. I created a repository with a test case that follows the installation instructions and creates a basic test: https://github.com/deviantintegral/nightwatch-accessibility-test

Using node 14:

$ node node_modules/.bin/nightwatch --env chrome test.js
[Test] Test Suite
=================
ℹ Connected to localhost on port 9515 (2913ms).
  Using: chrome (96.0.4664.55) on Mac OS X platform.

✔ Running Test:

✔ Element <body> was visible after 25 milliseconds.
✔ Passed [ok]: Elements must only use allowed ARIA attributes
✔ Passed [ok]: ARIA role should be appropriate for the element
✔ Passed [ok]: aria-hidden='true' must not be present on the document body
✔ Passed [ok]: Required ARIA attributes must be provided
✔ Passed [ok]: ARIA roles used must conform to valid values
✔ Passed [ok]: ARIA attributes must conform to valid values
✔ Passed [ok]: ARIA attributes must conform to valid names
✔ Passed [ok]: Inline text spacing must be adjustable with custom stylesheets
✔ Passed [ok]: Buttons must have discernible text
✔ Passed [ok]: Elements must have sufficient color contrast
✔ Passed [ok]: IDs of active elements must be unique
✔ Passed [ok]: IDs used in ARIA and labels must be unique
✔ Passed [ok]: id attribute value must be unique
✔ Passed [ok]: Headings should not be empty
✔ Passed [ok]: Heading levels should only increase by one
✔ Passed [ok]: Links with the same name must have a similar purpose
✔ Passed [ok]: Images must have alternate text
✔ Passed [ok]: Alternative text of images should not be repeated as text
✔ Passed [ok]: Banner landmark should not be contained in another landmark
✔ Passed [ok]: Contentinfo landmark should not be contained in another landmark
✔ Passed [ok]: Main landmark should not be contained in another landmark
✔ Passed [ok]: Document should not have more than one banner landmark
✔ Passed [ok]: Document should not have more than one contentinfo landmark
✔ Passed [ok]: Document should not have more than one main landmark
✔ Passed [ok]: Ensures landmarks are unique
✔ Passed [ok]: Links must have discernible text
✔ Passed [ok]: <ul> and <ol> must only directly contain <li>, <script> or <template> elements
✔ Passed [ok]: <li> elements must be contained in a <ul> or <ol>
✔ Passed [ok]: Interactive controls must not be nested
✔ Passed [ok]: All page content should be contained by landmarks
✔ Passed [ok]: Elements should not have tabindex greater than zero
✖ Failed [fail]: (id attribute value must be unique [<div class="block block-entity-browser-block block-entity-browser-blockcontent-browser-modal" id="block-entity-browser-blockcontent-browser-modal">] https://dequeuniversity.com/rules/axe/4.3/duplicate-id?application=axeAPI)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)


FAILED: 1 assertions failed and  32 passed (1.87s)
_________________________________________________

TEST FAILURE:  1 assertions failed, 32 passed (5.301s)

 ✖ test
 – Test (1.87s)

    POST  /timeouts/async_script - ECONNREFUSED
Error: connect ECONNREFUSED 127.0.0.1:9515
 Error while running .setTimeoutsAsyncScript() protocol action: Error ECONNREFUSED: connect ECONNREFUSED 127.0.0.1:9515

    POST  /execute_async - ECONNREFUSED
Error: connect ECONNREFUSED 127.0.0.1:9515
 Error while running .executeScriptAsync() protocol action: Error ECONNREFUSED: connect ECONNREFUSED 127.0.0.1:9515

On our actual project, we get a slightly different error, but this could be because this is running on linux inside of a Docker container, while the above was running on my macOS host:

✔ Passed [ok]: svg elements with an img role have an alternative text
✔ Passed [ok]: Elements should not have tabindex greater than zero
✖ Failed [fail]: (All page content should be contained by landmarks [<h2 id="primary-tabs-title" class="visually-hidden">Primary tabs</h2>] https://dequeuniversity.com/rules/axe/4.3/region?application=axeAPI)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)


FAILED: 1 assertions failed and  48 passed (3.85s)
_________________________________________________

TEST FAILURE:  1 assertions failed, 48 passed (9.02s)

 ✖ nightwatch/example.nightwatch
 – Demo test (3.85s)

 Error while running .setTimeoutsAsyncScript() protocol action: Unknown command 404 Not Found

 Error while running .executeScriptAsync() protocol action: Unknown command 404 Not Found

✖ Failed [fail]: (aXe failed to execute)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)

 Error while running .setTimeoutsAsyncScript() protocol action: Unknown command 404 Not Found

 Error while running .executeScriptAsync() protocol action: Unknown command 404 Not Found

This may be a regression of #23.

Error if CWD is not node_modules' parent directory

{ Error: ENOENT: no such file or directory, open '/app/test/ui/node_modules/axe-core/axe.min.js'
    at Object.fs.openSync (fs.js:653:18)
    at Object.fs.readFileSync (fs.js:554:33)

Nightwatch is running from /app/test/ui, but node_modules is in /app/node_modules.

You could use https://www.npmjs.com/package/find-node-modules here instead

const source = fs.readFileSync(path.resolve(path.join('node_modules', 'axe-core', 'axe.min.js')), 'utf8')```

Error when no results

I have markup, which aXe cli considers valid. No errors. However, when using with nightwatch accessibility plugin the test fails due to no results returned by Axe code. Which is currently considered as error in code of this plugin

nightwatch-accessibility compatibility with nightwatch 2.x

Hello,

I noticed that #160 was opened from dependabot to try upgrading nightwatch to the next major version. When I try using the current latest version of nightwatch-accessibility with nightwatch 2.x it fails with "aXe failed to execute" with no more information logged about what happened.

I do understand that error message comes from here:

node-nightwatch-accessibility/assertions/accessibility.js

Line 30 in 0ffdd41

this.api.assert[negate ? 'ok' : 'fail']('aXe failed to execute')

What steps are recommended when it comes to compatibility with nightwatch 2.x?