Wikimedia Quality and Test Engineering Team/Playbooks/Accessibility

QA General information

Definitions

Accessibility is the design of products, devices, services, vehicles, or environments so as to be usable by people with disabilities. The concept of accessible design and practice of accessible development ensures both "direct access" (i.e. unassisted) and "indirect access" meaning compatibility with a person's assistive technology (for example, computer screen readers).

Accessibility can be viewed as the "ability to access" and benefit from some system or entity. The concept focuses on enabling access for people with disabilities, or enabling access through the use of assistive technology; however, research and development in accessibility brings benefits to everyone.

https://en.wikipedia.org/wiki/Accessibility

Web accessibility, or eAccessibility, is the inclusive practice of ensuring there are no barriers that prevent interaction with, or access to, websites on the World Wide Web by people with physical disabilities, situational disabilities, and socio-economic restrictions on bandwidth and speed. When sites are correctly designed, developed and edited, more users have equal access to information and functionality. https://en.wikipedia.org/wiki/Web_accessibility

Web accessibility testing is a subset of usability testing where the users under consideration have disabilities that affect how they use the web. The end goal, in both usability and accessibility, is to discover how easily people can use a web site and feed that information back into improving future designs and implementations.

Play 1 Understand why testing accessibility is an important part of the QA process

Imagine a world in which every single human being can freely share in the sum of all knowledge. About 15% of the world's population lives with some form of disability. Therefore, the Quality and Test Engineering team, along with the development teams, must ensure people with disabilities are able to interact with Wikimedia products. That is to say, that they can perceive, understand, navigate, and contribute.

Play 2 Identify the places where accessibility testing is required (scope of accessibility testing)

We can be involved in accessibility testing in a number of places, for example:

Reviewing UX designs
Reviewing/writing user stories
Encouraging developers to use automated accessibility checking tools (e.g. aXe, WAVE)
Testing small changes or bug fixes
Testing when everything has been developed (e.g. on Production)
Checklist of testing items depending on the stage of development.

Working on a brand new project or feature

Where you might have the time and opportunity to perform accessibility testing at various points, such as reviewing UX designs, user stories, setting up automated accessibility checks, performing exploratory testing against a large number of items listed in WCAG21 and types of disability.

Testing a bug fix or functionality change

Where you might only have time to perform accessibility testing targeting a small number of potential accessibility issues identified as highest risk.

Play 3 Learn the WMF Design Style Guide and Principles

The Wikimedia Design Style Guide ensures a consistent look and behavior for our products. Our interfaces are our brand. These principles and guidelines aim to help designers and developers with their Wikimedia projects, as part of the Foundation or in a volunteer capacity. The style guide features unique focus areas like accessibility, internationalization, and localization.

Also, according to Wikimedia's Accessibility Statement, the Foundation aims to support WCAG 2.1 AA guidelines. Make sure you read them carefully and incorporate them to your daily testing procedures.

Play 4 Explore possible accessibility testing strategies/approaches/techniques

There are different strategies to testing accessibility. In this section we’ll outline 5 approaches.

Coverage-based
1. Coverage based on items in WCAG21
2. Coverage based on types of disability
Risk-based
1. Which functionality is likely to cause users the most problems?
2. For an/each area, identify what kinds of accessibility issues it will most likely have
3. For an/each WCAG21 item, identify which areas are mostly likely to have that particular issue
Activity-based
1. Follow a script or checklist depending on the activity or product you are testing. (See play 8)
  - e.g. https://www.w3.org/WAI/test-evaluate/preliminary/
  - e.g. Testing checklist for Abstract Wikipedia
2. Run accessibility checking tools (e.g. aXe, WAVE)
3. Perform exploratory testing (possibly using assistive technologies, e.g. screen readers)
4. Test as particular personas
  - e.g. https://www.w3.org/WAI/people-use-web/user-stories/ and https://www.w3.org/WAI/people-use-web/abilities-barriers/
Evaluation-based
1. Automated accessibility checks (e.g. aXe, perhaps run in CI)
2. Comparison with WCAG21 guidelines (quickref) and techniques
People-based
1. Testing by end-users
2. Testing by accessibility experts

Examples

VisualEditor watchlist functionality

When saving an edit with VisualEditor, the user has an option to add it to their watchlist for a particular amount of time. It had been mentioned that the UI controls which allow a user to do this might have accessibility problems. I wanted to do some accessibility testing following the playbook above. I give a report below, identifying the techniques I used.

I only had a limited amount of time to test this. Therefore, I took a focused approach.

First, I used Firefox’s default accessibility checker (available via devtools). This found two issues related to the watchlist elements, but I consider them to be false positives (the failure was “Clickable elements must be focusable and should have interactive semantics”, but I believe we are following all the recommendations mentioned there). This is using the technique 4.1 mentioned above.

Then, I used the aXe Firefox extension. This found no accessibility issues with the watchlist elements. Again, following technique 4.1.

I then took a risk-based approach to identify the accessibility issues the functionality is most likely to have (technique 2.2). I thought being able to access the UI controls using only a keyboard and having those controls read correctly by a screen reader were most likely.

For keyboard only use, I tested interacting using only a keyboard (technique 3.4, taking on the persona of a user who cannot use a mouse). I found I could do everything with a keyboard I could do with a mouse. The tabbing order was logical. I was able to achieve my task.

For screen readers, I checked that we were following the correct WCAG21 guidelines (technique 4.2). I assumed that guideline 1.1 (Text Alternatives) was the correct guideline we should be following to make it possible for screen readers to read our UI. I identified that the checkbox (“Watch this page”) is using technique H44 and using a <label> element to identify it. The dropdown is following technique ARIA9, using elements with the attribute “aria-labelledby”. These are marked by WCAG as techniques “sufficient” to ensure we have successfully met guideline 1.1.

To make sure, I could have installed a screen reader and checked that it read the UI controls correctly, but I did not deem it necessary.

Colour contrast of IPInfo tool

See phab:T309828#8206171. Use of technique 4.2 and WCAG21 technique G18.

Could also have used technique 3.2 as accessibility checking tools like aXe can find content with insufficient colour contrast.

Play 5 Explore the documentation and useful links

Note: Here could be the links to QA documentation, e.g. test planning, test instructions (test scripts, API calls, queries, step-by-step instructions, test cases/test results reports etc.)

Play 6 Communicate the importance of accessibility to the development team from an early stage of your engagement

Communicate to the development team the role of accessibility in the product development cycle and reference this playbook. Although accessibility testing might be a given for QAs, it is a good practice to make this point explicit to the development team from the beginning of the engagement.

Play 7 Give feedback to the development team regarding accessibility issues

Establish a tracking document for accessibility issues. The purpose of this document is to provide an overview of the testing results, and the number of issues.
Make sure you report accessibility issues by filing a phabricator ticket, tagging the responsible teams and adding the “Accessibility” project tag

Play 8 Create an Accessibility checklist for every product you are testing

If the product you are testing does not have a checklist dedicated to testing accessibility, start one. Creating a checklist will help you maintain consistency in your testing. In addition, if a new tester takes over the work, the checklist ensures the best practices will follow or improve.

You can use the webpage Easy Checks – A First Review of Web Accessibility to start a new checklist.

Play 9 Update and improve documentation accordingly

Document your testing practices, checklists, techniques, etc. and publish them in https://www.mediawiki.org/wiki/Wikimedia_Quality_and_Test_Engineering_Team and add them to the Test_Artifact category. Making them available for everyone will contribute to knowledge transfer.