Implement a junit reporter

[pgrond]

We are using Bitbucket for running pipelines, and test reporting in Bitbucket only supports Junit format output. There is a pa11y Junit reporter but not one for pa11y-ci.
Is it possible to implement this? Or use the pa11y one? I could not find any documentation on how to use pa11y reporters for pa11y-ci.

[aarongoldenthal]

Hi @pgrond,

At the moment trying to use pa11y reporters with pa11y-ci results in no output (details are captured in #174).

Just my opinion, but it is definitely possible, although with some challenges. As as example, here's a (very) quick adaptation of pa11y-reporter-junit as a pa11y-ci reporter (this could definitely be improved, but it produced a report that validated against the JUnit XSD):

'use strict';

// Converted to a pa11y-ci reporter from pa11y-reporter-junit at
// https://github.com/macieklewkowicz/pa11y-reporter-junit/blob/master/lib/reporter.js.

const builder = require('junit-report-builder');

// The `results` method is called once for each page that is tested and
// includes the URL and test results. Only accessibility issues are
// included, not all checks.
const results = (results) => {
    const suite = builder.testSuite().name(results.pageUrl);
    results.issues.forEach((issue) => {
        suite
            .testCase()
            .className(issue.code)
            .name(`[${results.pageUrl}] ${issue.selector}`)
            .failure(`${issue.message}\n\nContext: ${issue.context}`);
    });
};

// The `error` method is called when an error occurs executing pa11y
// against a page.
const error = (message) => {
    console.error(message);
};

// The `afterAll` method is called after all pages have been tested.
const afterAll = () => {
    const junitReport = builder.build();
    console.log(junitReport);
};

module.exports = {
    results,
    error,
    afterAll
};

To run you'd need to npm install junit-report-builder, save the above code in a local file (for example, ./pa11y-ci-reporter-junit.js), and run by pointing the --reporter options to the local file and piping the output to a file (I don't speak enough Bitbucket to translate better than that).

pa11y-ci --reporter ./pa11y-ci-reporter-junit.js https://pa11y.org https://github.com/pa11y/pa11y > junit.xml

One challenge with these results is that pa11y-ci only reports accessibility issues that are identified, not all checks that are performed, so the results are something like:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="https://pa11y.org/" tests="0" failures="0" errors="0" skipped="0"/>
  <testsuite name="https://github.com/pa11y/pa11y" tests="127" failures="127" errors="0" skipped="0">
    <testcase classname="WCAG2AA.Principle3.Guideline3_2.3_2_2.H32.2" name="[https://github.com/pa11y/pa11y] #query-builder-test-form">
      <failure message="This form does not contain a submit button, which creates issues for those who cannot submit the form using the keyboard. Submit buttons are INPUT elements with type attribute &quot;submit&quot; or &quot;image&quot;, or BUTTON elements with type &quot;submit&quot; or omitted/invalid.&#xA;&#xA;Context: &lt;form id=&quot;query-builder-test-form&quot; action=&quot;&quot; accept-charset=&quot;UTF-8&quot; method=&quot;get&quot;>&#xA;  &lt;query-builder data-target=&quot;...&lt;/form>"/>
    </testcase>
    <testcase classname="WCAG2AA.Principle4.Guideline4_1.4_1_2.H91.Textarea.Name" name="[https://github.com/pa11y/pa11y] #feedback">
      <failure message="This textarea element does not have a name available to an accessibility API. Valid names are: label element, title undefined, aria-label undefined, aria-labelledby undefined.&#xA;&#xA;Context: &lt;textarea name=&quot;feedback&quot; class=&quot;form-control width-full mb-2&quot; style=&quot;height: 120px&quot; id=&quot;feedback&quot;>&lt;/textarea>"/>
    </testcase>
    ... (plus 125 more)
  </testsuite>
</testsuites>

So, it appears as no tests run if all pa11y tests pass. There have been requests to offer an option to list all tests in the results, including those that pass (pa11y/pa11y#687, pa11y/pa11y#700). It seems like having that first would be a more representative JUnit report.