How accurate is Alleviator?

W3C ACT Rules: does it get the right answer?

The Accessibility Conformance Testing (ACT) Rules are written by a W3C community group. Each rule comes with small example pages that must pass, must fail, or don't apply. A tool is consistent with a rule when it never passes a page that should fail, and never fails a page that should pass.

We load all 1,211 example pages into Alleviator's real scanner, at desktop and phone sizes, exactly as a scan does. W3C's own scoring tool then compares the results with the expected ones.

Alleviator is built on axe-core, the open-source engine many tools use, plus our own checks. The 2 extra rules come from those: our keyboard test, which presses Tab through the page to check that focus is visible, and checking inside a site's own frames. Of the 5 rules not covered, 3 need a person to judge, like whether a page title is descriptive, so they're in every report's manual checklist. The other 2 (orientation locked with CSS transforms, and names for summary elements) aren't matched to an ACT rule yet.

These results are our own run of W3C's tool. We've submitted them to W3C for its list of implementations, and they're not listed there yet. We compare with axe-core's listing because it's the engine we build on. Other fully automated tools on W3C's list have up to 33 consistent rules, such as SortSite and Total Validator.

Raw results: act-earl.json (W3C's EARL format, the one its scoring tool reads).

GDS audit: how many barriers does it find?

In 2017 the UK Government Digital Service built a set of test pages with 142 accessibility barriers and checked which ones each popular tool found. We run Alleviator on the same pages and judge each result the same way: a barrier only counts if the finding actually identifies it.

Read this fairly. The other tools' results are the ones recorded in the audit, mostly from 2017, and those tools have improved since. Counting errors only, SortSite and Tenon found more. Many of our finds are warnings, marked "needs review" in a report, because a person has to confirm them. A table with no header cells, for example, might be a layout table that doesn't need any.

Raw results, with every test and how we judged it: gds-results.json.

What no checker can tell you

Even the best result here leaves 76 of the 142 barriers unfound. Automated tools can't judge whether a description matches its image, whether the reading order makes sense or whether instructions are clear. That's why every Alleviator report includes a guided checklist for the WCAG criteria that need a person, and why we never say a scan alone makes a site compliant.

See which criteria we check automatically and which need a person in our WCAG 2.2 checklist.