salesforce / sa11y Goto Github PK

View Code? Open in Web Editor NEW

84.0 12.0 21.0 5.78 MB

Salesforce Automated Accessibility Testing Libraries

Home Page: https://www.npmjs.com/org/sa11y

License: BSD 3-Clause "New" or "Revised" License

JavaScript 10.67% TypeScript 87.94% HTML 1.24% Shell 0.14%

accessibility-testing accessibility-automation accessibility test automation salesforce javascript axe

sa11y's Introduction

Salesforce Accessibility Automation Libraries

Automated Accessibility Testing Libraries and Tools (@sa11y packages) based on axe-core providing support for Jest unit tests, WebdriverIO component/integration tests used by teams in Salesforce. However, they are not specific to Salesforce and can be used to test any UI supported by axe-core for accessibility. These libraries are designed to be flexible, customizable and reusable to support automated accessibility testing in different testing workflows from unit to integration tests.

Docs
- References
Packages

Docs

Developer Guidelines
- Refer to Quick start instructions to get started, if you are a developer looking to Contribute
Code of Conduct
Changelog
LICENSE

References

Packages

This repo contains the following packages for automated accessibility testing:

Jest integration

Provides a toBeAccessible() accessibility matcher for Jest
- integrates the assertAccessible API with the Jest assertion API
Provides an option to set up the sa11y API to be invoked automatically at the end of each test
To add accessibility testing to your Jest tests use this package

WDIO integration

Provides assertAccessible(), assertAccessibleSync() APIs that can be used with WebdriverIO to check accessibility of web pages rendered in browsers

assertAccessible API

Checks DOM or HTML Element for accessibility issues and throws an error if a11y issues are found
To add accessibility testing to your Javascript unit tests not using Jest, use this package

a11y results formatter

Formats raw JSON output of a11y issues from axe into an easy to consume format by consolidating and cross-referencing
Used by assert Accessible API and Jest a11y matcher
To use axe directly and want to format the results from axe.run use this package
A new formatter groupViolationResultsProcessor also made available to group the a11y violations per jest test case as the existing formatter would generate test failures for each violation

Preset accessibility rules

Provides Base, Extended, Full accessibility preset rules as axe configuration
The Base preset rule is used by default in the Jest a11y matcher and assert Accessible APIs
- The APIs can be overridden to use the Extended or Full ruleset as necessary

Minified bundle for testing in browser

Provides a minified version of selected @sa11y libraries to be injected into a browser (using webdriver) and executed from integration testing workflows.
Gives WCAG SC for rulesets in addition with axe tags

Internal packages

Test utilities

Private package providing test utilities for @sa11y packages

Integration Tests

Private package providing integration tests for @sa11y packages

Common

Common utilities, constants, error messages for @sa11y packages

Dependency graph

Epilogue

A wise person once said…

"Don't talk to me until I've had coffee and you've run axe"

(Image courtesy: @shleewhite, @jorycunningham)

sa11y's People

Contributors

Stargazers

Watchers

Forkers

mohanraj-r rickschmoo rahman123 isabella232 andyhaskell jorycunningham jhausler1266 nolanlawson jasonschroeder-sfdc jrosesfdc langstonbenny lakamson rakhithjk esnait icecityman navateja-alagam rraavi korede rconjoe

sa11y's Issues

jest-html-reporter >> Error: Invalid character (�) in string

When a test fails with a11y issues and jest-html-reporter is being used, there is an error about an "Invalid character". Investigate and fix the error.

jest-html-reporter >> Error: Invalid character (�) in string: Error: expect(1 issues)toBeAccessible: expected document to have no accessibility violations but found 1 issues:

 "⭕ (image-alt) Images must have alternate text: foo:nth-child(2),img"
        🔗 Help URL: https://dequeuniversity.com/rules/axe/3.5/image-alt(0 issues)

fix(jest): automatic checks should not be triggered for failed tests

Automatic check is triggered regardless of the test status which would result in the original test failure if any getting overwritten by a11y failures if any from automatic checks.
Tests using the sa11y jest api would get tested twice with automatic checks - once as part of the sa11y API in the test and again as part of the automatic check
- a11y issues from automatic checks would overwrite the a11y issues found by the API
- If the sa11y API has been added to the test to check specific intermediate states of the DOM, enabling automatic checks could result in missed a11y issues

Workarounds

Current

Enable automatic checks in a parallel CI workflow and not in the main CI workflow.

Future

With the default Jasmine test runner in Jest v26 there doesn't seem to be clean way to detect test failures from Jest's teardown hooks
But jest-circus offers event handlers which allows binding to test failure events among others
- jest-circus is the default test runner starting Jest v27. After Jest v27 gains more ground and known issues (#65) are fixed this could be a viable option
But even if the test failure status can be detected reliably, the question then is if the automatic checks can be triggered from within the test context - from within a CustomEnvironment that extends JSDOM env e.g. where the event handler is bound
- With custom reporter / results processor e.g., accessing DOM is not possible and the test context is no longer available (#64).

doc: create a Pull request template

3 types: Feature request, Bug, Question
That encourages creating a Issue and discussing the changes before making them
- and reference the issue in the PR
If a release is to be made to npm with the PR
- if versions and changelog needs to be updated etc
Chore: update dependencies

Support for latest Node LTS Version (v20)

Hi,
it would be great if you could please upgrade and provide support for the latest Node LTS version (v20). We are currently trying to update our projects, but are receiving the following warnings for this package during installation:

At first glance, it also looks as if everything is working correctly apart from the warnings during installation.

Fix build failure in code coverage step on merge to master

The code coverage github action which has been working fine on #7 failed with following error once #7 was merged to master.

https://github.com/salesforce/sa11y/runs/572119038

Code Coverage Report0s
##[error]Cannot read property 'head' of undefined
Run romeovs/[email protected]
TypeError: Cannot read property 'head' of undefined
at main$1 (/home/runner/work/_actions/romeovs/lcov-reporter-action/v0.2.17/dist/main.js:22938:41)
##[error]Cannot read property 'head' of undefined

Looking at the src from stacktrace romeovs/lcov-reporter-action/v0.2.17/dist/main.js:22938:41, guess this could be because of the squash and merge into master. Don't see issues enabled for romeovs/lcov-reporter-action - no other resources linked in Readme to report issues either.

Possible fix might be to just use a different github action for code coverage.
- As part of the switch also explore if the action can offer more control in terms posting code cov details in PR (e.g. only post to PR if code cov has changed, instead of repeatedly posting the same info every time there is a push) and more control over email notifications.
- Another option might be to post code coverage info in a different place e.g. as a badge in Readme, rather than in the PR for every push. That will reduce the noise and also put the info in a more visible location.

test(wdio): migrate from ts-node for wdio tests

Currently ts-node is used in wdio conf as described in TypeScript Setup · WebdriverIO

But babel with '@babel/preset-typescript' is already used for jest tests as described in Getting Started · Jest

Can we reuse babel instead as is it much more popular and actively maintained

Also ts-node seems to be resulting in some issues as described below

Extended/full rulesets cause Pango-WARNING messages

Running tests with the extended or full rulesets when there's text present causes WARNING messages from Pango to be displayed, that don't appear when using the base ruleset. Unsure as to whether these warnings may interfere with testing any particular rule.

Environment:

node v18.15.0
@salesforce/sfdx-lwc-jest v1.3.0
@sa11y/[email protected]

Testing Component:
Markup:

<template>
  <lightning-card>
    <button></button>
    <p>Some text</p>
  </lightning-card>
</template>

Javascript:

import { LightningElement } from "lwc";

export default class TestSa11y extends LightningElement {}

Test:

import { createElement } from "lwc";
import TestSa11y from "c/testSa11y";

describe("c-test-sa11-y", () => {
  it("Tests button-name", () => {
    const element = createElement("c-test-sa11-y", {
      is: TestSa11y
    });
    document.body.appendChild(element);

    expect(1).toBe(1);
  });
});

Test Results:
Using extended ruleset (process.env.SA11Y_RULESET = "extended" in setupEnvVars.js) (test result file)
Using base ruleset ((process.env.SA11Y_RULESET = "base" in setupEnvVars.js) (test result file)

Expected Behaviour: Changing rulesets should not bring up undesired warnings clogging up CI/full run logs.

Add ability to register a11y matcher in jest config once per project

Motivation

The a11y matcher API has to be registered before its used. Currently this needs to be done in each test module as part of test setup as shown below.

import { registerA11yMatchers } from '@sa11y/jest';
beforeAll(() => {
    registerA11yMatchers();
});
describe('integration test @sa11y/jest', () => {
    it('should have a11y matchers working', () => {
        expect(expect.toBeAccessible).toBeDefined();
        expect(document).toBeAccessible();
    });
});

This could be made less tedious by registering the a11y matchers once in the jest config of the consuming project and then all the tests in the project can use the a11y API without any additional setup.

For example a consuming project should be able to register a11y matchers by extending the config as below in the jest.config.js at the project root:

const { jestConfig } = require('@sa11y/jest');

module.exports = {
    ...jestConfig,
    // Rest of the Project's Jest config ..
};

Resources

Similar setup can be found in

Feature request: add support for node@18 (current LTS)

Hi team, could you upgrade sa11y so that it supports the latest Node LTS version (v18)?

For reference, when installing a project that contains sa11y with node@18, you get these warnings:

npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE   package: '@sa11y/[email protected]',
npm WARN EBADENGINE   required: { node: '^14 || ^16' },
npm WARN EBADENGINE   current: { node: 'v1[8](https://github.com/pozil/sfdc-ui-lookup-lwc/actions/runs/4162909628/jobs/7202651044#step:5:9).13.0', npm: '9.3.0' }
npm WARN EBADENGINE }
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE   package: '@sa11y/[email protected]',
npm WARN EBADENGINE   required: { node: '^14 || ^16' },
npm WARN EBADENGINE   current: { node: 'v18.13.0', npm: '[9](https://github.com/pozil/sfdc-ui-lookup-lwc/actions/runs/4162909628/jobs/7202651044#step:5:10).3.0' }
npm WARN EBADENGINE }
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE   package: '@sa11y/[email protected]',
npm WARN EBADENGINE   required: { node: '^14 || ^16' },
npm WARN EBADENGINE   current: { node: 'v18.13.0', npm: '9.3.0' }
npm WARN EBADENGINE }
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE   package: '@sa[11](https://github.com/pozil/sfdc-ui-lookup-lwc/actions/runs/4162909628/jobs/7202651044#step:5:12)y/[email protected]',
npm WARN EBADENGINE   required: { node: '^14 || ^16' },
npm WARN EBADENGINE   current: { node: 'v18.13.0', npm: '9.3.0' }
npm WARN EBADENGINE }
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE   package: '@sa11y/[email protected]',
npm WARN EBADENGINE   required: { node: '^14 || ^16' },
npm WARN EBADENGINE   current: { node: 'v18.[13](https://github.com/pozil/sfdc-ui-lookup-lwc/actions/runs/4162909628/jobs/7202651044#step:5:14).0', npm: '9.3.0' }
npm WARN EBADENGINE }

From my experience, sa11y works just fine despite the warnings.

Button with shadow DOM contents reports "Buttons must have discernible text"

For a DOM structure like so:

<button>
  <div>
    #shadow-root
      Click me!
  </div>  
</button>

The sa11y Jest plugin will report Buttons must have discernible text, even though the button has text – it's just in the shadow DOM.

Here is a reduced test case: nolanlawson@0825dff

To run it, you can check out the code and then run npx jest ./packages/assert/.

Full error output

FAIL  packages/assert/__tests__/assert.test.ts
  assertAccessible API
    ✓ should trigger axe runtime exception for non existent rule (8 ms)
    ✕ should throw no errors for dom with no a11y issues with config 0 (130 ms)
    ✕ should throw no errors for dom with no a11y issues with config 1 (69 ms)
    ✕ should use default document, ruleset, formatter when called with no args - expecting 0 assertion (54 ms)
    ✓ should use default document, ruleset, formatter when called with no args - expecting 1 assertion (46 ms)
    ✓ should throw an error with a11y issues found for dom with a11y issues (24 ms)
    ✕ should not throw error with HTML element with no a11y issues (44 ms)
    ✓ should throw error with HTML element with a11y issues (19 ms)
    ✓ should test audio without timing-out using src 0 (4 ms)
    ✓ should test audio without timing-out using src 1 (4 ms)
    ✓ should test audio without timing-out using src 2 (5 ms)
    ✓ should test video without timing-out using src 0 (6 ms)
    ✓ should test video without timing-out using src 1 (7 ms)
    ✓ should test video without timing-out using src 2 (9 ms)

  ● assertAccessible API › should throw no errors for dom with no a11y issues with config 0

    A11yError: 1 Accessibility issues found
     * (button-name) Buttons must have discernible text: #shadow-button
        - Help URL: https://dequeuniversity.com/rules/axe/4.3/button-name

      62 |         }
      63 |         if (a11yResults.length > 0) {
    > 64 |             throw new A11yError(violations, a11yResults, opts);
         |                   ^
      65 |         }
      66 |     }
      67 |

      at Function.checkAndThrow (packages/format/src/format.ts:64:19)
      at assertAccessible (packages/assert/src/assert.ts:44:15)
      at packages/assert/__tests__/assert.test.ts:45:9

  ● assertAccessible API › should throw no errors for dom with no a11y issues with config 1

    A11yError: 1 Accessibility issues found
     * (button-name) Buttons must have discernible text: #shadow-button
        - Help URL: https://dequeuniversity.com/rules/axe/4.3/button-name

      62 |         }
      63 |         if (a11yResults.length > 0) {
    > 64 |             throw new A11yError(violations, a11yResults, opts);
         |                   ^
      65 |         }
      66 |     }
      67 |

      at Function.checkAndThrow (packages/format/src/format.ts:64:19)
      at assertAccessible (packages/assert/src/assert.ts:44:15)
      at packages/assert/__tests__/assert.test.ts:45:9

  ● assertAccessible API › should use default document, ruleset, formatter when called with no args - expecting 0 assertion

    expect(received).resolves.toHaveLength(expected)

    Expected length: 0
    Received length: 1
    Received array:  [{"description": "Ensures buttons have discernible text", "help": "Buttons must have discernible text", "helpUrl": "https://dequeuniversity.com/rules/axe/4.3/button-name?application=axeAPI", "id": "button-name", "impact": "critical", "nodes": [{"all": [], "any": [{"data": null, "id": "button-has-visible-text", "impact": "critical", "message": "Element does not have inner text that is visible to screen readers", "relatedNodes": []}, {"data": null, "id": "aria-label", "impact": "serious", "message": "aria-label attribute does not exist or is empty", "relatedNodes": []}, {"data": null, "id": "aria-labelledby", "impact": "serious", "message": "aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty", "relatedNodes": []}, {"data": {"messageKey": "noAttr"}, "id": "non-empty-title", "impact": "serious", "message": "Element has no title attribute", "relatedNodes": []}, {"data": null, "id": "presentational-role", "impact": "minor", "message": "Element's default semantics were not overridden with role=\"none\" or role=\"presentation\"", "relatedNodes": []}], "html": "<button id=\"shadow-button\"><div id=\"shadow-button-inner\"></div></button>", "impact": "critical", "none": [], "target": ["#shadow-button"]}], "tags": ["cat.name-role-value", "wcag2a", "wcag412", "section508", "section508.22.a", "ACT"]}]

      55 |             document.body.innerHTML = testDOM;
      56 |             expect.assertions(expectedAssertions);
    > 57 |             await expect(getViolationsJSDOM()).resolves.toHaveLength(expectedViolations);
         |                                                         ^
      58 |             await assertAccessible().catch((e) => checkA11yError(e));
      59 |         }
      60 |     );

      at Object.toHaveLength (node_modules/expect/build/index.js:198:20)
      at packages/assert/__tests__/assert.test.ts:57:57

  ● assertAccessible API › should not throw error with HTML element with no a11y issues

    expect(received).toHaveLength(expected)

    Expected length: 0
    Received length: 163
    Received string: "1 Accessibility issues found
     * (button-name) Buttons must have discernible text: #shadow-button
        - Help URL: https://dequeuniversity.com/rules/axe/4.3/button-name"

      58 |     } finally {
      59 |         if (expectNoError) {
    > 60 |             expect(err.message).toHaveLength(0);
         |                                 ^
      61 |         } else if (expectRuntimeError) {
      62 |             expect(err.message).toContain(axeRuntimeExceptionMsgPrefix);
      63 |         } else {

      at checkA11yErrorFunc (packages/test-utils/src/utils.ts:60:33)
      at Object.<anonymous> (packages/assert/__tests__/assert.test.ts:72:9)

Test Suites: 1 failed, 1 total
Tests:       4 failed, 10 passed, 14 total
Snapshots:   3 passed, 3 total
Time:        1.47 s, estimated 2 s
Ran all test suites matching /.\/packages\/assert\//i.

I'm not sure where exactly this is being thrown, because the latest version of the Axe extension does not seem to have this issue:

fix: missing pkg dependency of @sa11y/common

@sa11y/common is listed as a dev dependency of @sa11y/format because initially only common types declared in that pkg were being used in Typescript.
But currently non-type declarations from the common pkg are being used (e.g. errMsgHeader). This results in error when trying to install @sa11y/format on its own.

The pkg dependency checker in place has failed to catch this.

Fix the missing dependency
Install all @sa11y pkgs in isolation and fix any dependency issues
Automate this check as much as possible

build: remove semantic-release dependency

semantic-release was added as a dependency recently to publish a github release
- but couldn't get it to work as desired
it brings in a huge list of dependencies and non-trivial learning curve
it is not mono-repo aware/friendly
when lerna is used for versioning and publishing, semantic-release doesn't offer a whole lot over it
the github release notes needs to be hand-edited from the changelog anyways

Source of truth for rulesets

From #3

What is the source of truth for this list? Or asked another way, how do we know when this list is stale and we need need to update it?

We have an internal doc that we maintain with the product accessibility team (@cordeliadillon & Jesse ). Currently making sure the rules here are in sync with the doc is a manual process. I have been also wondering about how this could be automated to make sure it is updated.

Versioning the doc itself and maintaining a changelog in the doc along with current deployment status might help.
Or to ease maintenance and sync overhead could we make this repo the single source of truth and deprecate the doc @cordeliadillon ?

Unpin axe-core for compatibility with new versions

I recently updated axe-core to 4.8.2 in a project, but because the @sa11y/common package pins axe-core to 4.7.0, I end up with two distinct copies of axe-core in my project which don't have compatible types (see dequelabs/axe-core#4081). This causes TypeScript to complain:

    Type 'import("/home/runner/work/PrairieLearn/PrairieLearn/node_modules/axe-core/axe").Result' is not assignable to type 'import("/home/runner/work/PrairieLearn/PrairieLearn/node_modules/@sa11y/common/node_modules/axe-core/axe").Result'.
      Types of property 'nodes' are incompatible.
        Type 'import("/home/runner/work/PrairieLearn/PrairieLearn/node_modules/axe-core/axe").NodeResult[]' is not assignable to type 'import("/home/runner/work/PrairieLearn/PrairieLearn/node_modules/@sa11y/common/node_modules/axe-core/axe").NodeResult[]'.
          Type 'import("/home/runner/work/PrairieLearn/PrairieLearn/node_modules/axe-core/axe").NodeResult' is not assignable to type 'import("/home/runner/work/PrairieLearn/PrairieLearn/node_modules/@sa11y/common/node_modules/axe-core/axe").NodeResult'.
            Types of property 'target' are incompatible.
              Type 'UnlabelledFrameSelector' is not assignable to type 'string[]'.
                Type 'CrossTreeSelector' is not assignable to type 'string'.
                  Type 'ShadowDomSelector' is not assignable to type 'string'.

This library should almost definitely unpin axe-core to allow new minor versions to be used without @sa11y/common needing to explicitly change.

I'd be happy to make this change if a maintainer could confirm that it will be accepted. It's not clear to me why the @sa11y/* packages pin dependencies in the first place.

Build failure with Jest snapshot errors

Build is failing with Jest snapshot errors saying they are not up-to-date - but the updated snapshots have been checked in and the tests pass locally after a clean build. Even checked out the branch to a diff clean location in my machine and tried out the CI commands - no issues 😕

Originally posted by @mohanraj-r in #13 (comment)

chore: find a better way to track TODOs in code

Trevor: Should we capture these TODOs as issues?

Thanks Trevor, Yes that is a good idea.

I have been thinking of exploring GitHub Discussions to capture TODOs so that Issues are focussed on bugs etc. What do you think about that?
Looks like there are also tools to convert TODOs in code to issues automatically e.g. TODO to Issue · Actions · GitHub Marketplace · GitHub. I will explore them more.

Originally posted by @mohanraj-r in #54 (comment)

How to ignore certain rules for certain tests

Hi,
I am wondering if we can turn off some certain rules for specific tests. For example, there are some accessibility errors happening due to the third-party usage. It is out of our control to fix those errors. We want to skip that check if it is possible

 expect(1 issues)toBeAccessible: expected document to have no accessibility violations but found 1 issues:

     "⭕ (aria-allowed-attr) Elements must only use allowed ARIA attributes: #popover-trigger-\\:rb\\:"
    	🔗 Help URL: https://dequeuniversity.com/rules/axe/4.7/aria-allowed-attr
    	🔗 WCAG Criteria: SA11Y-WCAG-SC4.1.2-P1(0 issues)

      130 |     })
      131 |     // To test using all rules provided by axe
    > 132 |     await expect(container).toBeAccessible(full)
          |                             ^
      133 | })
      134 |
      135 | /**

Something like jest-axe config? https://www.npmjs.com/package/jest-axe#setting-global-configuration

fix(jest): codecov, automatic check tests to work with Jest v27

Upgrading from Jest v26 to v27 results in test failures from automatic check tests that exercise the custom hook. Investigate and fix. (Found in #62)

 ● automatic checks registration › should run when called directly without setup

    Hooks cannot be defined inside tests. Hook of type "afterEach" is nested within "should run when called directly without setup".

    > 68 |         afterEach(async () => {
         |         ^
      69 |             await automaticCheck(opts);
      70 |         });
      71 |     }

      at eventHandler (node_modules/jest-circus/build/eventHandler.js:107:11)
      at registerSa11yAutomaticChecks (packages/jest/src/automatic.ts:68:9)
      at Object.<anonymous> (packages/jest/__tests__/automatic.test.ts:37:9)

spike: preset-rule configurations based on dynamic criteria

Would it make sense to have preset-rules by

a rough estimate of how easy the fix would be
- e.g. image-redundant-alt is usually a quick fix while aria-hidden-body is often a bit more involved.
based on performance by benchmarking individual rules
- e.g. a fast/basic preset-rules for use in performance critical scenarios
WCAG levels - A, AA etc ?
- Might not have value for most of our UIs, as we require both for GA features, but could be useful for beta/pilot UIs.

doc(jest, wdio): add screenshots, GIF demos to jest, wdio packages

Add screenshot, demo video (using asciinema) showing errors, illustrating usage etc

feat(wdio): add screenshot, URL a11y issues output from @sa11y/wdio

Create a template for a11y issues output from webdriver tests which includes additional info such as screenshot and URL of page tested. Save screenshot locally (to a temp dir) and include path to image. Reuse (and simplify) existing template from Java webdriver tests : https://github.com/mohanraj-r/sa11y/tree/WIP/add_template_formatter

import { assertAccessible, assertAccessibleSync } problem

Error
The test runs fine when not using @sa11y but once its been added as imprt, below error comes up

import { assertAccessible, assertAccessibleSync } from '@sa11y/wdio';
^

SyntaxError: Unexpected token {
at Module._compile (internal/modules/cjs/loader.js:872:18)

Package.js
{
"name": "accessiblity_testing",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo "Error: no test specified" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC",
"devDependencies": {
"@sa11y/jest": "^0.1.5-beta.0",
"@sa11y/wdio": "^0.1.2-alpha.0",
"@wdio/cli": "^6.7.4",
"@wdio/firefox-profile-service": "^6.6.0",
"@wdio/local-runner": "^6.7.3",
"@wdio/mocha-framework": "^6.7.3",
"@wdio/spec-reporter": "^6.7.0",
"@wdio/sync": "^6.7.3",
"chromedriver": "^87.0.0",
"wdio-chromedriver-service": "^6.0.4"
},
"dependencies": {
"bufferutil": "^4.0.2",
"utf-8-validate": "^5.0.3"
}
}

Test case
import { assertAccessible, assertAccessibleSync } from '@sa11y/wdio';

const LoginPage = require('../pageobjects/login.page');
const SecurePage = require('../pageobjects/secure.page');

describe('My Login application', () => {
it('should login with valid credentials', () => {
LoginPage.open();

    LoginPage.login('tomsmith', 'SuperSecretPassword!');
    assertAccessible();
});

});

Missing deliverable code for @sa11y/[email protected]

Hi we're using @sa11y/jest in the LWC Recipes sample app and updating to v4.0.0 breaks the import of the Sally Jest matchers:

import { registerSa11yMatcher } from '@sa11y/jest';

Can you document the upgrade process to v3 to v4?

S@11y test results - how do we handle the nested interactive elements?

Started testing S@11y on one of our Salesforce instances - got the following results. Not sure how to proceed in fixing these items.

I understand that scaling to 500% would be addressed by the reflow work, so ok in discounting those results. But how do we fix the nested interactive elements?

Ran query: brazil-build test:local - It's customized in our code package

S@11y results below:

[Chrome 113.0.0.0 darwin #0-0] Error: expect(received).toEqual(expected) // deep equality
[Chrome 113.0.0.0 darwin #0-0]
[Chrome 113.0.0.0 darwin #0-0] Expected: [Error]
[Chrome 113.0.0.0 darwin #0-0] Received: [A11yError: 10 Accessibility issues found
[Chrome 113.0.0.0 darwin #0-0] * (link-name) Links must have discernible text: .action-link
[Chrome 113.0.0.0 darwin #0-0] - Help URL: https://dequeuniversity.com/rules/axe/4.3/link-name·
[Chrome 113.0.0.0 darwin #0-0] * (nested-interactive) Ensure interactive controls are not nested: .slds-is-disabled
[Chrome 113.0.0.0 darwin #0-0] - Help URL: https://dequeuniversity.com/rules/axe/4.3/nested-interactive·
[Chrome 113.0.0.0 darwin #0-0] * (nested-interactive) Ensure interactive controls are not nested: .branding-favorites-list-button
[Chrome 113.0.0.0 darwin #0-0] - Help URL: https://dequeuniversity.com/rules/axe/4.3/nested-interactive·
[Chrome 113.0.0.0 darwin #0-0] * (nested-interactive) Ensure interactive controls are not nested: .slds-global-actions__help
[Chrome 113.0.0.0 darwin #0-0] - Help URL: https://dequeuniversity.com/rules/axe/4.3/nested-interactive·
[Chrome 113.0.0.0 darwin #0-0] * (nested-interactive) Ensure interactive controls are not nested: div[data-aura-rendered-by="\31 37\:220\;a"]
[Chrome 113.0.0.0 darwin #0-0] - Help URL: https://dequeuniversity.com/rules/axe/4.3/nested-interactive·
[Chrome 113.0.0.0 darwin #0-0] * (nested-interactive) Ensure interactive controls are not nested: .menuTriggerLink
[Chrome 113.0.0.0 darwin #0-0] - Help URL: https://dequeuniversity.com/rules/axe/4.3/nested-interactive·
[Chrome 113.0.0.0 darwin #0-0] * (nested-interactive) Ensure interactive controls are not nested: .slds-global-actions__notifications
[Chrome 113.0.0.0 darwin #0-0] - Help URL: https://dequeuniversity.com/rules/axe/4.3/nested-interactive·
[Chrome 113.0.0.0 darwin #0-0] * (nested-interactive) Ensure interactive controls are not nested: .branding-userProfile-button
[Chrome 113.0.0.0 darwin #0-0] - Help URL: https://dequeuniversity.com/rules/axe/4.3/nested-interactive·
[Chrome 113.0.0.0 darwin #0-0] * (meta-viewport-large) Users should be able to zoom and scale the text up to 500%: meta[name="viewport"]:nth-child(2)
[Chrome 113.0.0.0 darwin #0-0] - Help URL: https://dequeuniversity.com/rules/axe/4.3/meta-viewport-large·
[Chrome 113.0.0.0 darwin #0-0] * (meta-viewport-large) Users should be able to zoom and scale the text up to 500%: #vp
[Chrome 113.0.0.0 darwin #0-0] - Help URL: https://dequeuniversity.com/rules/axe/4.3/meta-viewport-large]
[Chrome 113.0.0.0 darwin #0-0] at checkA11yErrorWdio (/Users/yuanhu/workspace/ProservecrmTestAutomation/src/ProservecrmTestAutomation/dist/ui-test/accessibilityWdio.test.js:27:21)
[Chrome 113.0.0.0 darwin #0-0] at runMicrotasks ()
[Chrome 113.0.0.0 darwin #0-0] at async Context. (/Users/yuanhu/workspace/ProservecrmTestAutomation/src/ProservecrmTestAutomation/dist/ui-test/accessibilityWdio.test.js:39:9)

accesskeys rule not failing as expected

I have a simple LWC(c-hello-world) as below:

<template>
  <a href="google.com" accesskey="g">Link to Google</a>
  <a href="github.com" accesskey="g">Link to GitHub</a>
</template>

and my jest test looks like below:

import { createElement } from 'lwc';
import HelloWorld from 'c/helloWorld';
import { setup } from '@sa11y/jest';
import { extended, full } from '@sa11y/preset-rules';

describe('c-hello-world', () => {
  beforeAll(() => {
    setup();
  });
  it('duplicate access keys', async () => {
    const element = createElement('c-hello-world', {
      is: HelloWorld
    });
    document.body.appendChild(element);

    await expect(element.shadowRoot.querySelector('a')).toBeAccessible();
    await expect(element.shadowRoot.querySelector('a')).toBeAccessible(extended);
    await expect(element.shadowRoot.querySelector('a')).toBeAccessible(full);
  });
});

Surprisingly, this doesn't fail. I was expecting it to fail as per preset rule accessKeys mentioned here.

What is wrong with my test?

Node version: v14.19.3
@sa11y/jest: 3.1.0
@salesforce/sfdx-lwc-jest: 1.1.0
OS: macOS Monterey

EDIT: The above code actually works and it fails with error A11yError: 1 Accessibility issues found as expected. But once I add afterEach to my jest test, it stops failing and shows no accessibility issues.

This below code doesn't throw error as expected but shows as all passed.

import { createElement } from 'lwc';
import HelloWorld from 'c/helloWorld';
import { setup } from '@sa11y/jest';

describe('c-hello-world', () => {
  beforeAll(() => {
    setup();
  });

  afterEach(() => {
    // The jsdom instance is shared across test cases in a single file so reset the DOM
    while (document.body.firstChild) {
      document.body.removeChild(document.body.firstChild);
    }
    // Prevent data saved on mocks from leaking between tests
    jest.clearAllMocks();
  });

  it('duplicate access keys', async () => {
    const element = createElement('c-hello-world', {
      is: HelloWorld
    });
    document.body.appendChild(element);

    await expect(element.shadowRoot.querySelectorAll('a').length).toBe(2);
    await expect(element.shadowRoot.querySelector('a')).toBeAccessible();
  });
});

Why afterEach method is causing sa11y to not work as expected?

build: replace ncu with yarn upgrade

npm-check-updates is currently being used to update outdated dependencies.
Looks like yarn upgrade --latest could do the same
But it currently doesn't work on workspaces in a mono-repo
- Once this issue is fixed, replace ncu with yarn upgrade --latest

test(integration): create a integration test with LWC component(s)

Create barebones LWC components from the public OSS packages and

Verify that for non-accessible components appropriate a11y issues are thrown
Verify that accessible components throw no a11y error
Put the component in its various possible states (e.g. expanded etc) and check for a11y
- Use for doc/illustration

Extended/full rulesets cause "Not implemented: window.computedStyle(elt, pseudoElt)" Error

Running tests using the extended or full rulesets when some components are present, causes multiple Not implemented: window.computedStyle(elt, pseudoElt) errors to be shown via console.error. This does not happen with the base ruleset.

Environment:

node v18.15.0
@salesforce/sfdx-lwc-jest v1.3.0
@sa11y/[email protected]

Testing Component:
Markup:

<template>
  <form action="#">
    <input type="button" id="fail1InputButtonName" />
  </form>
</template>

Javascript:

import { LightningElement } from "lwc";

export default class TestSa11y2 extends LightningElement {}

Test:

import { createElement } from "lwc";
import TestSa11y2 from "c/testSa11y2";

describe("c-test-sa11-y2", () => {
  afterEach(() => {
    while (document.body.firstChild) {
      document.body.removeChild(document.body.firstChild);
    }
  });

  it("Tests input-button-name", async () => {
    const element = createElement("c-test-sa11-y2", {
      is: TestSa11y2
    });
    document.body.appendChild(element);

    await expect(element).not.toBeAccessible();
  });
});

Test Results:
Using extended ruleset (process.env.SA11Y_RULESET = "extended" in setupEnvVars.js) (test result file)
Using base ruleset (process.env.SA11Y_RULESET = "base" in setupEnvVars.js) (test result file)

Expected Behaviour: Changing rulesets should not cause errors.

ci: add checks that are performed on staged changes to ci

Checks that are performed on staged changes using lint-staged

can be bypassed
are localized only to the staged changes

Would be good to add the checks globally to be run in CI for consistency to catch any issues that were bypassed in local checkin.

Migrate from ts-jest to babel typescript transformer

From #2

Pros

With Jest 24 there's much better support for Typescript
The maintainer of ts-jest has basically said to just use native Jest: kulshekhar/ts-jest#961 (comment)
worth considering for the future since other projects have reported a speed improvement moving away from ts-jest.
Babel is already a dependency of Jest so technically it's already pulled it in to the project as a transitive dependency.

caveats

Eval the limitations and tradeoffs
- some features might not be available in babel yet
- caveats
Tried replacing ts-jest with babel/preset-typescript as per https://jestjs.io/docs/en/getting-started. Ran into some errors with jest trying to run the .js files as well as .d.ts files in addition to .ts files. Probably needs more config than what is mentioned in those docs.

Install fails on node version 18?

Is there a reason the packages are flagged at ^14 || ^16?

Node 14 EOL is in April, and node 16 EOL is later this year in October. ref

fix(format): error when expect(elem).not.toBeAccessible is called on an elem with no a11y issues

For a HTML element that has no a11y issues:
Expected: expect(elem).not.toBeAccessible() to throw a jest exception error
Got:

  TypeError: Cannot read property 'format' of undefined

      at message (../node_modules/@sa11y/jest/dist/matcher.js:57:137)
      at getMessage (../node_modules/expect/build/index.js:177:15)
      at processResult (../node_modules/expect/build/index.js:295:25)
      at asyncResult.then.aResult (../node_modules/expect/build/index.js:353:28)

This should not be a critical issues as the not matcher is not expected to be used with the toBeAccessible matcher in primary workflows.

spike: Metrics collection

How to collect, report and visualize/alert on metrics ?

Evaluate

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Pending Approval

These branches will be created by Renovate only once you click their checkbox below.

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

chore(deps): update dependency semantic-release to v23

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

github-actions

.github/workflows/codeql-analysis.yml

actions/checkout v3.6.0@f43a0e5ff2bd294095638e18286ca9a3d1956744

github/codeql-action v2.24.10@ffd3158cb9024ebd018dbf20756f28befbd168c7

github/codeql-action v2.24.10@ffd3158cb9024ebd018dbf20756f28befbd168c7

github/codeql-action v2.24.10@ffd3158cb9024ebd018dbf20756f28befbd168c7

.github/workflows/nodejs.yml

actions/checkout v3.6.0@f43a0e5ff2bd294095638e18286ca9a3d1956744

actions/setup-node v3.8.2@1a4442cacd436585916779262731d5b162bc6ec7

codecov/codecov-action v3.1.6@ab904c41d6ece82784817410c45d8b8c02684457

actions/checkout v3.6.0@f43a0e5ff2bd294095638e18286ca9a3d1956744

actions/setup-node v3.8.2@1a4442cacd436585916779262731d5b162bc6ec7

.github/workflows/scorecards.yml

actions/checkout v3.6.0@f43a0e5ff2bd294095638e18286ca9a3d1956744

ossf/scorecard-action v2.3.1@0864cf19026789058feabb7e87baa5f140aac736

actions/upload-artifact v3.1.3@a8a3f3ad30e3422c9c7b888a15615d19a852ae32

github/codeql-action v2.24.10@ffd3158cb9024ebd018dbf20756f28befbd168c7

npm

package.json

@babel/core 7.24.4

@babel/preset-env 7.24.4

@babel/preset-typescript 7.24.1

@commitlint/cli 17.8.1

@commitlint/config-conventional 17.8.1

@semantic-release/changelog 6.0.3

@semantic-release/exec 6.0.3

@tsconfig/node14 1.0.3

@types/jest 28.1.8

@types/node 15.14.9

@typescript-eslint/eslint-plugin 5.62.0

@typescript-eslint/parser 5.62.0

@wdio/cli 7.36.0

@wdio/jasmine-framework 7.33.0

@wdio/local-runner 7.36.0

@wdio/spec-reporter 7.33.0

babel-jest 28.1.3

chromedriver 115.0.1

commitizen 4.3.0

conventional-changelog-cli 2.2.2

cspell 6.31.3

depcheck 1.4.3

doctoc 2.2.1

eslint 8.57.0

eslint-config-prettier 8.10.0

eslint-import-resolver-typescript 3.6.1

eslint-plugin-eslint-comments 3.2.0

eslint-plugin-import 2.29.1

eslint-plugin-jest 27.9.0

eslint-plugin-markdown 3.0.1

eslint-plugin-notice 0.9.10

eslint-plugin-prettier 4.2.1

eslint-plugin-tsdoc 0.2.17

eslint-watch 8.0.0

husky 8.0.3

is-ci 3.0.1

jest 28.1.3

jest-environment-jsdom 28.1.3

jest-jasmine2 28.1.3

jest-junit 16.0.0

lerna 6.6.2

lint-staged 13.3.0

lockfile-lint 4.12.1

markdown-link-check 3.11.2

package-dependency-graph 1.14.4

prettier 2.8.8

rimraf 3.0.2

semantic-release 19.0.5

ts-node 10.9.2

typescript 4.9.5

vertioner 1.0.6

wdio-chromedriver-service 7.3.2

webdriverio 7.36.0

node ^16 || ^18

packages/assert/package.json

axe-core 4.8.3

@jest/globals 28.1.3

node ^16 || ^18

packages/browser-lib/package.json

axe-core 4.8.3

@rollup/plugin-commonjs 23.0.7

@rollup/plugin-node-resolve 15.2.3

@rollup/plugin-replace 5.0.5

rollup 3.29.4

rollup-plugin-polyfill-node 0.13.0

rollup-plugin-progress 1.1.2

rollup-plugin-sizes 1.0.6

rollup-plugin-terser 7.0.2

rollup-plugin-typescript2 0.36.0

node ^16 || ^18

packages/common/package.json

axe-core 4.8.3

webdriverio 7.36.0

node ^16 || ^18

packages/format/package.json

axe-core 4.8.3

@jest/globals 28.1.3

node ^16 || ^18

packages/jest/package.json

@jest/test-result ^27

jest-matcher-utils ^27

@jest/globals 28.1.3

jest >=27.0.0

node ^16 || ^18

packages/preset-rules/package.json

@jest/globals 28.1.3

axe-core 4.8.3

markdown-table-ts 1.0.3

node ^16 || ^18

packages/test-integration/package.json

@jest/globals 28.1.3

node ^16 || ^18

packages/test-utils/package.json

@jest/globals 28.1.3

node ^16 || ^18

packages/wdio/package.json

axe-core 4.8.3

webdriverio 7.36.0

webdriverio >=6.0.0

node ^16 || ^18

Check this box to trigger a request for Renovate to run again on this repository

fix: errors with node lts v16

Following error occurs when using node v16
Had to switch to node v14 temporarily in #81 as a workaround
Fix the error and revert back to using lts

$ yarn test:clean
...
$ package-dependency-graph --graphviz --svg docs/sa11y_dependency_graph.svg
node:internal/modules/cjs/loader:1183
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: The module 'sa11y/node_modules/canvas/build/Release/canvas.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 83. This version of Node.js requires
NODE_MODULE_VERSION 93. Please try re-compiling or re-installing
the module (for instance, using `npm rebuild` or `npm install`).
    at Object.Module._extensions..node (node:internal/modules/cjs/loader:1183:18)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (sa11y/node_modules/canvas/lib/bindings.js:3:18)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
  code: 'ERR_DLOPEN_FAILED'
}
error Command failed with exit code 1.

test(lint): upgrade to typescript-eslint v3.x, typescript v4.x and fix newly identified issues

Release v3.0.0 of typescript-eslint/typescript-eslint · GitHub includes breaking changes.
Upgrading to v3.x.x results in many errors/warnings in current codebase.
Upgrade to v3 and fix new issues caught in code

Incorrect Readme displayed at repo root

The file at .github/Readme.md is being displayed at the repository root suppressing the README at root level. This could be by special convention of .github folder. Fix this!

build: improve versioning workflow

Current workflow is to bump versions whenever there a package changes from master (using "lint:version": "vertioner && yarn lerna exec --since master vertioner")

@trevor-bliss pointed out couple of issues with this approach:

There can be version conflicts between PRs. Imagine there are several PRs open at the same time and they all bump the version to the next patch version. As soon as one gets merged, all other PRs must go in and bump their version again. This potentially repeats for every open PR when a different PR is merged.

There may be versions in github without corresponding NPM releases. Unless we release every commit to master to NPM, there may be several version bumps before a new version in NPM. Not necessarily a bad thing but can be confusing IMO.

We put the burden of versioning on the PR author. When I contribute to other projects I just focus on the code fix rather than worrying about whether something should be a patch vs. minor release.

Just my opinion but I think it makes more sense to leave the versions untouched in PRs and then do a release from master or a release branch and we as project maintainers decide when that release happens and what version to bump to.

Look into alternate solutions that resolves these issues.