w3c / wai-wcag-em-report-tool

2021 redesign of the WCAG-EM Report Tool

WCAG-EM Report Tool

Preview

About this tool

The WCAG-EM Report tool is an assistant that takes the user through the process of evaluating the accessibility of a website, using the WCAG Evaluation Methodology.

It is developed by the Web Accessibility Initiative (WAI) at the World Wide Web Consortium (W3C). Initially developed with support from the WAI-ACT Project, and updated with support of the WAI-Tools Project, co-funded by the European Commission (EC).

Environments

This tool is available on:

Environment	URL(s)	Hosted at
production	w3.org/WAI/eval/report-tool	GitHub Pages of this repository
preview of main branch	https://wcag-em-report-tool-2021-redesign.netlify.app/	Netlify
previews of Pull Requests	see comment at individual Pull Request	Netlify

Local development

For local development, you will need to have Node installed.

We use Svelte so that we can express the code of this project in components and Rollup to bundle all JavaScript together.

First time

1. Clone the repository (you can fork it first)
2. In the project folder, run npm install, this gets all the Node dependencies

After first time

1. For development, run npm run dev, this builds the project, serves the application on a local server (localhost:10001 per default) so that you can look at it and watches for any changes you make to the code.
2. To only build the application, run npm run build, this builds the project without serving it.

Publishing in a sub folder

If you want to publish this tool somewhere that is not the root domain (eg w3.org/WAI/eval/report-tool instead of wcagemreporttool.com), set the BASEPATH variable before you run npm run build|dev, for example:

export BASEPATH="/WAI/eval/report-tool"

This ensures routing and paths to assets continue to work.

Contributing

If you find any problem, feel free to file an issue.

You can also add a new language, see How to add a language.

wai-wcag-em-report-tool's Issues

Structured Report link seems wrong

[!!]
Location: Overview Page / About This Tool

The link to "Structured Report" goes to the current report tool View Report page. Should that link be going somewhere else, like to the new View Report page?

Not all data imported (in “Open Report” functionality)

When using test report VA, the following are not imported:

• conformance target of that report, 'AA' is not copied over, in the conformance dropdown in Step 1, it is set to 'A', in the Your Report box and View Report page it is displayed as “ WAI:WCAG2AA-Conformance“
• author
• date
• technologies that are not in the default list are not copied over (DOM, ECMAScript 5, HTML5)
• sample pages

Test old reports on compatibility

Old reports have a different JSON format and within the import-fuction all kinds of things will give errors because some keys don't exist in exports from the old tool

Tweak “Show info” content

(From BJ)

In Step 2: Explore the Target Website / Section: Web Technologies Relied Upon, it explains how to add, but the mechanism has changed. This needs updating to reflect how it works now.

In Step 2: Explore the Target Website / Section: Web Technology: CSS and HTML are listed as examples, but they are already there by default.

Use scope attribute in result tables

MW:

[ED-med] In the "Detailed audit results" tables, it seems the "Success Criterion" cells would be a row header (scope="row"), particularly to help distinguish the "Results" cells in the larger tables. The WAI guidance on tables (https://www.w3.org/WAI/tutorials/tables/one-header/) also recommends designating "scope" for larger tables (e.g., also setting scope="col" for the column headers); I didn't have an issue with VoiceOver but V.O. may have been compensating.

Parse evaluation specifics as Markdown?

Step 5, Report Findings, has a a field “Record of evaluation specifics”.

In some test reports, like VA, evaluators a list. Example content:

- Chrome, versie 91
- Firefox, versie 89
- Edge, versie 91
- Acrobat PRO DC

If we would parse this as Markdown it would get list markup. It would be unlikely to harm reports.

Question about reports that are in progress

If someone has started a report in the old tool and needs to finish it,

1. will they be able to open the report in the new tool to finish it? and if not...
2. will the URL change of the old tool affect them being able to import their report and continue/finish it?

Would hate for someone to lose a report in progress during this transition. Want to know if someone will be able to continue their work in progress. Thanks!

Show your report button transparency and z-index

When creating a report and hiding the “Your report” summary on the right side, a button “Show your report” becomes available.
The button has a fixed position when scrolling down the page.

Because the button is transparent, it mingles with other text on its path.
It also goes behind the “Import data EARL/JSON” button.

Schermopname.2021-08-12.om.21.01.44.mov

Step 2 - adding technologies slightly confusing

The combination of the button label "Add other Web Technology" and the spacings/separation between the fields "Web Technology", "Specification Address", and the button make it unclear. In particular, for most other fields it is sufficient to simply add the text that will appear in the report. But for these fields you have to press the add other technology button to get your current text into the report. We should probably consider making the "show info" text more explanatory too.

Summary counter not correct with samples

Some criteria are counted multiple times when using multiple samples. Probably a counting issue that can be fixed with a filtering

[Redesign] Do we want “Detailed audit results“ in “Step 5”?

Step 4 - Show full description link

[ED - Med] Step 4: Audit Sample

Each Success Criteria has a "Show full description" link. For the longer success criteria this is great because you can activate the button to show the full language.

Many success criteria don't have any more detail in the description than what is already loaded on screen by default. The button is still there and doesn't do anything which can be confusing. Can that button be removed or hidden in the cases when it is not needed because all the detail is already showing?

Steps 3 & 4 - default labels inconsistent

The default labels on Step 3 are "structured_sample_1", "structured_sample_2", "random_sample_1", "random_sample_2", ... Yet these show as "Sample 1", "Sample 2", "Sample 3", "Sample 4", ... in Step 4.

First, I suggest using the term "Page" instead of "Sample" because the sample is the collection of multiple pages.

Secondly, if technically possible, the default labels should be the same on both steps.

Store the evaluation with local storage

(Original request by Eric Eggert)
Adding a local storage feature may make it easier for users to leave the project unfinished and to work offline.

Create zip file on release

cc @julezrulez

We have done this in other tools in the past and could add it in the GitHub Action that runs on release.

Ctrl +s to save functionality does not work

I recall some discussion on this although I can't remember the resolution. Having just completed an audit using the existing reporting tool, this functionality is essential! Especially considering the issues with local storage, working on multiple projects, working with multiple auditors.

Not sure if this has been removed or simply does not work in Edge. Currently when I press ctrl+s I get the webpage save dialog which will save the HTML not the JSON.

If HTML is used in observations, it is not escaped

Some reports, like VA, may say things like:

Use a <h2> element

Is there a way we could escape that HTML, while still parsing HTML that is meant as HTML? Or perhaps we should parse Markdown but not HTML (which it is normally a superset of)?

Report SD is also a good example of where this causes unwanted results, it breaks the results table.

Filters on Audit sample page don't translate well

WCAG version:
The text "Added in" is hardcoded

Conformance target:
The translations are set in a variable on page load, so if you switch language on the audit sample page itself, they won't update. Only after navigating to another page and navigating back they will be re-initialized and translated correctly

🚀 Deploy the redesign

Making the switch in code

• Ensure routing works (can look at how it is done in ATAG Report Tool); the tool will likely live on a sub URI of w3.org/WAI, we should make sure paths continue to work done
• Will we reverse proxy to GitHub Pages like we do for ATAG Report Tool, or some other mechanism? yes

Comms

• Add banner to old tool announcing date some time before done

Document the format we expect to work with “Import EARL as JSON”

cc @nitedog

There's currently no way to know:

• as a user, if “Import as EARL” will work with a given tool.
• as a vendor of automated accessibility tooling, if “Import as EARL” will work with my tool or what I need to do to make that happen

Add numbers to “Detailed audit results”

On the View Report page, there are some numbers that summarise the audit results, detailing how many SCs fall into each category.

These numbers are missing on the "Detailed audit results" part of the Report Findings page.

The task in this issue is to add them there, too.

update "changelog" for users and messaging

Background: Here is a little about purpose of changelogs -- although that's mostly for translations whereas this changelog is about informing regular users of the tool.

https://github.com/w3c/wai-wcag-em-report-tool/wiki/Changelog:-What's-new-in-the-2021-redesign-of-the-WCAG-EM-Report-Tool

I think the audience for that was mostly EOWG reviewers. Now we are using it for users.

Consider if edits would make it better meet the needs of users?

(Also if you can write it to apply now and also when we deploy, e.g., change
"...the differences between what's currently live and what is in the new 2021 version."
-> "...the differences between Version 2 and Version 3."

Also I'd like a few bullets &/or sentences summary for the announcements. Here's a reminder that Hidde is planning to add that to the top of the changelog to benefit users, and Shawn will pull from that for messaging.

p.s. ftr. I would slightly prefer the changelog be like the others, in a WAI page. Although it probably doesn't matter (and maybe we should start changing to using GitHub wiki pages instead)?

expand/collapse all sections during step 4

Would be nice to have the option of collapsing/expanding all the WCAG criteria during step 4 to reduce the page length.

Document translations

• Find out how translations work and what is needed: which files does someone need to change/update?
• Check what's still relevant from translations.md, move any relevant content to the wiki
• Clean up translations.md
• Document how translations can be submitted (what does a PR for a translation look like) in How to add a translation
• Make documentation easy to find (maybe as an Issue template and in the Wiki, with README pointing at the Wiki)

(example of issue template implementation)

How to deal with older reports that contain evaluations for criteria that are out of scope?

In the old tool, you could add observations for criteria that were not in scope, such as observations WCAG 2.1 criteria in a report scoped to WCAG 2.0.

The new tool does not allow for that.

What do we do when importing an old report into the new tool, that has more observations than the conformance target?

Proposed solution: add warning when this is the case:

You're importing a report that evaluates more criteria than its conformance target allows. This could be due to the WCAG version and/of conformance level. To use the report in this tool, increase the conformance target in the JSON file.

Position sticky on table headers in View Report leave margin in Safari

CD:

• On Safari Version 14.1.1 (16611.2.7.1.4), on the "Report Findings" and "View Report" pages, the table headers do not stick to the top of the view port when scrolling, leaving a margin that shows the table content above and below the header. This does not happen on Chrome and Firefox.

Screen.Recording.2021-06-25.at.14.23.10.mov

Filename depending on website name

Currently with Cltr+S the file is saved end gets the name 'evaluation.json'

In the old report tool when saving a file the filename gets the 'Website name' prefixed (all in lower case and spaces and strange characters turned into dashes). When doing a lot of audits or doing more audits at the same time your download folder gets filled with saved files. By adding the website name to these file names you could easily distinguish the different audits.
So for my website named "200 OK" at 200ok.nl the file name would be: 200-ok-evaluation.json

Sample title out of bounds when to long

When you add a sample with a very long title it will go out of bounds and it won't get snapped. When using a long title with spaces, this is not a problem.

See image:

[ED-low] Links to Step 5 do not move view to top of viewport

Describe the bug
Following the links to Step 5 do not move view to top of viewport.
The links to other steps move the viewport to the top. The links to step 5 (from step 4 or step 6) do not.

To Reproduce
Steps to reproduce the behavior:

1. Go to Step 4 or Step 6
2. Click on 'Report Findings' in the navigation bar at the bottom of the page
3. Step 5 load but the viewport does not move to the top

Expected behavior
For every other step the viewport moves to the top. It should move also in this step.

Desktop (please complete the following information):

• OS: macOS Big Sur
• Browser: Safari
• Version 14.1.1 (16611.2.7.1.4)

Overview of audit results - data entry?

[!!]
Location: View Report page

On the View Report page, the fourth heading down is "Overview of audit results." I cannot find where that data would be entered. I have looked through all the steps and filled in all of the data fields (except all of the success criteria outcomes and observations) and can't seem to get that heading populated in the view report page. All it says is "Not Provided." There is no "edit" button on that section either to edit it directly on that page.

Am I missing something? If that is no longer being used for data, then that heading needs to be removed.

Support multiple results per assertion

A single assertion can have multiple results when the same test is applied on one page multiple times. Tools generating results like this should be supported.

Add acknowledgements page

Like https://www.w3.org/WAI/atag/report-tool/acknowledgements

version banners

Here is a proposal. Feel free to implement as is, or suggest changes.

Style:

• yellow banner at the very top, e.g., https://wai-website-theme.netlify.app/components/colors/
• text weight normal (not bold)

Now

Text for existing/old tool:

A redesign of this WCAG-EM Report Tool will be published here in August or September 2021. You can see the Redesign Preview and changelog of revisions.

When publish

Old tool banner text:

An updated redesign of this WCAG-EM Report Tool is available at https://www.w3.org/WAI/eval/report-tool/ (changelog of revisions).

New tool banner text:

The previous version of this WCAG-EM Report Tool is available at @@ (changelog of revisions).

Sticky Your Report block can overlap results on Report Findings page

Add the possibility to also enter Recommendations/Tips

Add the possibility to give more feedback in the report. For example about issues that are not covered by the WCAG level tested for, but have an impact on the accessibility of the site.

Like remarks about heading structure or semantics. But also to report bugs in the software or give compliments to the developers for work nicely done.

I add such recommendations with all my reviews and most of the time these issues also end up on the items-to-be-fixed list.

Some titles are cut off in Dutch translation on View Report page

It seems like something goes wrong with the processing of these translations as they get cut off after whitespaces

See image:

Use color to indicate results on audit page

In the old report tool when entering issues in Step 4 it was visually easy to recognize the different outcome. Green=success, Red=Fail. While 50 SC's is a long list scrolling though all succes criteria the color makes it easier to recognize the one you where searching for.

Could there also put a color to the checked succes criteria?

Cleanup warnings

There are some warnings that we might be able to get rid of quite easily.

On opening the tool:

After starting the tool in the console:

Add test files for “Import EARL as JSON” to wiki

@nitedog is asking around to get test files from various vendors so that we can:

• test the import functionality
• find out what the minimum requirements are for an import file to work

🌎 How to translate

Information on how to translate is available in the wiki: How to add a language

Add PDF to technologies

In the current list in Step 2 you could check several techniques. I always wondered why PDF is not in that list. Could PDF added to the list of check boxes?

I know you could do that by hand, but in the years that I do audits I have never added an technique by hand. Checkboxes are much easier.

Add WCAG-EM Report Tool version number to output JSON

(This was requested by someone who builds a tool that uses WCAG-EM Report Tool output)

Also show warning before performing 'Start new report' action

comment from @nitedog

we have a warning when you do 'Clear report', we should also show one when doing 'Start new report'

Report title not in output JSON

Let's add the report title to the output JSON

[ED-low] Consider not presenting the "Show full description" button when there is no extra description to show

On step 4, every Success Criteria includes a "Show full description" button after the first paragraph of the SC's description. Most SC only have one paragraph, and the button does not show additional content when activated. This might confuse users.
Consider not presenting the "Show full description" button when there is no extra description to show.

This is not exactly a bug, but there is no category to submit suggestions.

Use “Edit” as header for table columns with edit button

Comment from MW:

[ED-med] While the final "Detailed audit results" tables are marked up correctly, the last column doesn't have a table header. Most specialists I know (myself included) don't recommend empty table headers ('s), even if the wording is visually hidden. In this case, I think the word "Edit" would be really helpful visually as well as in the hidden screen reader text as I had to deduce what those icons would do and confirm it through clicking on them. While the programmatic names of the buttons include the column header (e.g., "Edit 1.1.1"), it still reads as a "blank" column name when first exploring the table with a screen reader which I would think is jarring. Also, the cells are blank when the final report is generated which might be okay for power users but confusing for customers reading the report (looks like an errand extra column).

'This page in' string may be difficult to understand

"This page in" may be difficult to understand for some. Why not use "Languages and Translations" as in other WAI templates?

Rename the tool to “WCAG Report Tool” (removing “-EM”)

Something along the lines of "WCAG-EM report Generator" or "WCAG Evaluation Assistant". The current name is confusing users about its purpose. This issue should be worked on together with the Education & Outreach Working Group.

Step 4 - add link to more info on EARL/JSON

Will you add a link from the "Add data from other (automated) tests" box to documentation on GitHub (or elsewhere) about this import functionality?

"Show info" buttons appear before their related fields in tab order

When navigating with the tab it may be distracting to have the "Show info" before the field itself. Should they appear before or after in tab order.