Prepending base URL

[loopy3025]

We are running pa11y through a bash script. We would like to run pa11y not only on ci, but locally, and on every environment (dev, stage, and prod). Currently, it seems I have to run pa11y using a different sitemap for each environment or, alternatively, run a separate pa11y test for each url in a loop and put in the base url with an argument for my bash function.

• Maintaining a pa11y test sitemap for each environment is not really sustainable.
• Running separate pa11y tests for each page does not report well. The report files don't survive the ci, so we are trying to write the results to a variable for publishing in our dev environment instead. There doesn't seem to be a great way to write the results to a variable either.

I'd like to be able to do one of three things:

• Call pa11y with both a base url and a sitemap of relative links I.E. npm run tests --sitemap sitemap.xml --baseurl https://baseurl.dev.com or,
• Call pa11y with the npm run tests $pagelist where $pagelist can be multiple pages instead of just one, or
• In a bash script (not js) write the pa11y run results to a variable instead of to the console or a file.

[loopy3025]

Never mind. I just found sitemap-find and sitemap-replace. Still, for some reason pa11y is checking the xml file instead of running it as a sitemap.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
  <url>
    <loc>https://oursite.docksal.site</loc>
  </url>
  <url>
    <loc>https://oursite.docksal.site/contact</loc>
  </url>
</urlset>

npm run tests --sitemap "$baseurl/themes/custom/$themefoler/pa11y/sitemaps/pa11y.xml" --sitemap-find https://oursite.docksal.site --sitemap-replace $baseurl

Running Pa11y on 2 URLs:
https://oursite.docksal.site/themes/custom/oursite/pa11y/sitemaps/pa11y.xml - 2 errors
https://oursite.docksal.site/ - 3 errors

It should be checking ousite.docksal.site and oursite.docksal.site/contact, but it's checking oursite.docksal.site and pa11y.xml. Am I calling for the sitemap wrong?

[loopy3025]

So it seems to not be using the --sitemap argument. It thinks the xml is a page. When I add an = sign in, I just get nothing instead.

npm run tests --sitemap=https://oursite.docksal.site/themes/custom/oursite/pa11y/pa11y-sitemap.xml

> [email protected] tests
> pa11y-ci

Running Pa11y on 0 URLs:

✔ 0/0 URLs passed

Adding -vvv seems to just give the version instead of verbose output.

Edit: looks like -d gets me verbose output but nothing seems to indicate why it's busted. I did fix/ensure my xml and it is valid.

[loopy3025]

I installed pa11y globally and tried running it with the pa11y-ci command instead of through npm.

pa11y-ci --sitemap https://oursite.docksal.site/themes/custom/oursite/pa11y/pa11y-sitemap.xml --sitemap-find https://oursite.docksal.site --sitemap-replace https://dev.oursite.com

With this response:

The sitemap "https://oursite.docksal.site/themes/custom/dev/pa11y/pa11y-sitemap.xml" could not be loaded

Which doesn't make much sense because my browser clearly displays the valid xml.

[josebolos]

Hi @loopy3025

The sitemap that you're trying to use doesn't seem to exist, or if it does, it's not publicly available:

$ curl -I "https://dev.docksal.site/themes/custom/dev/pa11y/pa11y-sitemap.xml"
curl: (28) Failed to connect to dev.docksal.site port 443 after 129492 ms: Connection timed out

I don't think your problem is related to pa11y.

[loopy3025]

It's not public. It's on my local. I have removed some of the parts of the URL because I don't want to expose these sorts of things publicly.

This is what's displayed in my browser at the xml location:

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
<url>
<loc>https://oursite.docksal.site</loc>
</url>
<url>
<loc>https://oursite.docksal.site/contact</loc>
</url>
</urlset>

However, you did help me out a bit. I CURLed the URL and got

curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I've never had an issue like this on this machine before, but then again I don't think I've ever tried parsing a local xml file through the command line through a url before.

It sure would be easier to specify a file location instead of a url.

At any rate, I'll push the file to the server and reference that one instead, I guess.

[loopy3025]

After working with this for a few days, I believe it would be a huge benefit if pa11y had a --run-local or --run-dev flag which would run the curl of the xml file as --insecure.

Here's my solution in case anyone is looking for a similar solution where they want to test during ci or manually on a website with multiple environments. This specific example is drupal on acquia cloud. This does not fail the test if pages fail, but does write a report.

.pa11yci.js:

/**
 * @file
 * Pa11y config.
 */

const isCI = process.env.CI;

module.exports = {
  defaults: {
    standard: 'WCAG2AA',
    hideElements: ['svg'],
    "viewport": {
      "width": 1280,
      "height": 1024
    },
    ignore: ['notice', 'warning'],
    "reporters": [
      ["cli"],
      ["pa11y-ci-reporter-html", { "destination": "../../../reports/pa11y-reports", "includeZeroIssues": true }]
    ],
    chromeLaunchConfig: {
      ignoreHTTPSErrors: true,
      args: ['--no-sandbox']
    }
  }
};

The Script .accessibility-test.sh

#!/bin/bash
# ----------------------
# accessibility-test.sh
# ----------------------
# Executes a pa11y command for any environment with defined sites
# 
# The site has one argument which is required, which is the site machine name in Acquia.
# 
# This script has 2 flags:
# -e is the environment. The options are local, ci, dev, stg, prod. Defaults to local. 
# -o is an override url which will use the full url you put in instead of building a url based on the site and environment. 
# 
# Notes: 
# - The override should be used if you are testing prod or an environment with a non-standard url.
# - Any pages at webny-protected-pages cannot be reached in prod.
# - dev and stage are set as the default Acquia machine name. If a different machine name has been set, then you need to use an override url.
# - in order to run this script on your local machine, you may need to globally install the pa11y packages with `npm -g install pa11y-ci pa11y-ci-reporter-html`
#
# Usage:
# Testing the dev environment on Acquia:
#   ./pa11y/.accessibility-test.sh thissite -e dev
# Testing the prod environment:
#   ./pa11y/.accessibility-test.sh thissite -e prod -o https://arts.ny.gov
# Testing a local Docksal environment with thissite.docksal.site as the url:
#   ./pa11y/.accessibility-test.sh thissite -e local
# Testing a local site with any other url:
#   ./pa11y/.accessibility-test.sh thissite -e local -o https://myweirdlocalurl.docksal
# ----------------------

# The acquia site machine name for the site we are testing
thissite=$1

# Set the username and password for shield
shieldcreds="shielduser:shieldpassword"

# By default, use the local environment
environment="local"

# get the environment flag and the override url, if any
while [[ $# -gt 0 ]]; do
  case $1 in
      -e|--environment) 
        environment=$2
        shift
        ;;
      -o|--overrideurl) 
        overrideurl=$2
        shift
        ;;  
      --default)
        DEFAULT=YES
        ;;
      -*|--*)
        echo "Unknown option $1"
        exit 1
        ;;
      *)
        POSITIONAL_ARGS+=("$1") # save positional arg
        shift # past argument
        ;;
  esac
done
set -- "${POSITIONAL_ARGS[@]}" # restore positional parameters

# If our user passed in an argument for the environment, use that instead
if [ $environment ] 
then 
  environment=$environment
fi

# Set the acquia url string
urlstring="https://${shieldcreds}@${thissite}${environment}.prod.acquia-sites.com"

# Get the base url for our environment
case $environment in
  "prod")
    # For prodm we use the base url
    if [ ! $overrideurl ]
    then 
      printf "ERROR: You should provide the full prod url with the \"-o\" flag if you are testing the prod environment.\n"
      exit 1
    fi
    ;;
  "stg")
    baseurl=$urlstring
    ;;
  "dev")
    baseurl=$urlstring
    ;;
  "ci")
    baseurl='http://127.0.0.1:8888'
    ;;
  *)
    # if the user put anything in besides an accepted environment, then use local instead
    printf "Environment not found. Defaulting to local. Please use prod, stg, dev, or ci if you don't want to use your local environment.\n\n"
    environment="local"
    baseurl="https://${thissite}.docksal.site"
    ;;
esac

# if there is an override url provided, set it as the base url instead
if [ $overrideurl ]; then baseurl=$overrideurl; fi

# The url of the sitemap
# TODO: By default, the sitemap in production should be the source of truth, not dev
sitemapurl="https://${shieldcreds}@${thissite}dev.prod.acquia-sites.com/pa11y/sitemap.xml"

# Colors
BOLDBLUE='\033[1;34m'
BLUE='\033[0;34m'
GREEN='\033[0;32m'
NC='\033[0m' # No Color

# Message the user
docksalmessage="${BLUE}Are you having trouble with the tester finding your local site? If your local site's url isn't formatted like \"https://mysite.docksal.site\", you will need to use the \"-o\" flag to specify the full url to your local environment. For example: -o https://mysite.docksal${NC}\n\n"
printf "${BOLDBLUE}Testing accessibility on the $environment environment at $baseurl.${NC}\n\n"
if [ $environment = "local" ]; then printf "$docksalmessage"; fi

# Test the pages
printf "${GREEN}\n=====================\nTest is using the sitemap at ${sitemapurl} to get its list of pages to test, but with the base url swapped out for the environment you designated (${baseurl}). ${NC}\n\n"

pa11y-ci --sitemap $sitemapurl --sitemap-find "https://${thissite}dev.prod.acquia-sites.com" --sitemap-replace $baseurl

printf "${BOLDBLUE}Accessibility test on the $environment $baseurl is complete. Don't forget that this script is just a first line of defense against accessibility regressions.${NC}\n\n"

if [ $environment = "local" ]; then printf "$docksalmessage"; fi

acquia-pipelines.yml during setup-env

- cd ${THEME_DIR} && npm cache clean --force && rm -rf node_modules && npm install
- source ${BLT_DIR}/scripts/pipelines/tests

blt.yml

command-hooks:
  frontend-test:
    dir: '${docroot}/themes/custom/oursite'
    command: 
      - 'npm run accessibility ${project.machine_name} -- -e ci'