Should you use accessibility testing tools?

By Olena Bulygina

Accessibility testing tools aim to help you improve the compliance of your site or app. But do they deliver on what they promise? And where do they fit in your accessibility testing toolkit?

There are tonnes of automated tools on the market, with wildly different claims. Some promise a passport to full compliance – others, merely some relief from your compliance headaches.

It’s a minefield. So to make sense of it all, I caught up with Dig Inclusion founder, Grant Broome, for CXcon: Rethink Accessibility – Inviqa’s online digital accessibility conference.

Together we explored the how, what, why, and when of accessibility testing tools and answered your questions live. 

Keep reading to get the key takeaways from our conversation. Or – if you have a little more time – you can check out the below video for the full talk.

What are automated accessibility testing tools?

Accessibility testing tools provide an automated way to check how well your website complies with the web content accessibility guidelines (WCAG).

They scan your website pages and check them against WCAG criteria, giving you a score or grade that helps you gauge your level of compliance. They also help you identify what you need to improve to make your digital product more compliant.

Google Lighthouse is an example of an automated accessibility testing tool. It provides a clear gauge of digital accessibility, using a tangible score that you can track and measure yourself against.

The benefits of accessibility testing tools 

Automated testing tools can be a great starting point for understanding how well your digital prototype or product complies with the web content accessibility guidelines. 

They’re especially handy if you need to scan a large volume of site pages and discover compliance issues quickly.

Let’s say you’re looking for missing image alt text. An automated tool can extract and serve you this information very quickly, while a manual check would take you ages.

Another great use case is checking forms. An automated tool can very quickly tell you whether your forms are missing labels, or are labelled twice. It’s a lot easier than manually checking your visual labels, and the ones lurking in your code and containers.

Automated testing improves and streamlines your manual testing – and it should be part of every accessibility tester's toolkit. 

Grant Broome

Accessibility testing tool examples

These are some of the automated tools I recommend and use daily in my accessibility work.

  • WAVE Evaluation Tool. This is a great browser plugin from WebAIM designed to support your manual evaluation and help you better understand accessibility issues. 
  • Google Lighthouse. This is a great feel tool that the whole team can use. You’ll need to go through the accessibility evaluations manually, but the accessibility scoring is very valuable.
  • Stark contrast checker. This contrast checker for Sketch helps you ensure that accessibility is baked into your digital product design.

The limitations of accessibility testing tools

Accessibility testing tools are super useful, but it’s important to understand their uses and limitations. 

Many of these tools are oversold as a complete solution to compliance. But this just isn’t the case. And relying on these tools as your only solution for testing accessibility is hugely limiting.

Here are the cons or limitations you should know about. 

Accessibility testing tools don’t guarantee compliance

Firstly, compliance does not guarantee accessibility.

It’s possible to build a web page with a perfect Lighthouse score that’s still not accessible. Only by testing with real users who experience disability can you understand whether your digital product is genuinely accessible.

The second key point here is that testing tools don’t guarantee compliance

Many tools are unclear on how their algorithms test your site, which WCAG version they test against (for example, WCAG 2.1), or even which WCAG criteria they test against.

If you’re experienced in accessibility and testing, you can do your due diligence and build a picture of how well a tool serves your needs. But it’s important to ensure you’re not relying on these tools alone to achieve compliance.

Testing tools can be unreliable

Different testing tools give you different results. You can pass a test with one tool and fail it using another. 

If you don’t have a background in accessibility, it’s difficult to know whether the results you receive from a tool are reliable or not. 

Decision making is limited

Testing tools augment our decision making around accessibility. But they don’t have the context and human experience that’s needed to make decisions. And accessibility testing involves a lot of considered decision making.

Take alt text. A tool can confirm the presence of image alt text. But it can’t say with any certainty whether the alternative text correctly describes the contents of an image. And it doesn’t understand context.

Yes, it can flag alt text that looks strange or might be incorrect. For example, where it includes a string of numbers. But it can’t actually decide for itself whether alt text is valid.

To illustrate the point, Inviqa partner and Dig Inclusion founder Grant Broome, tested the same image of a kitten using the Google accessibility tester.

First he checked the image with the alt text ‘kitten’. Then he tested the same image with the alt text ‘great white shark’.

Both versions scored 100% using the tool. The tool obviously doesn’t know its kittens from its sharks. But it also can’t assess the context of the image and deduce – for example – that the alt text describes a kitten that’s actually named ‘Great White Shark’.

It’s the perfect example of why automated testing requires human validation.

And there are other considerations. One tool rarely does everything, and you need to be as close as possible to the production code of your website.

How to use tools in your accessibility testing

So how do you navigate these factors and use tools to improve your accessibility testing?

Start by looking at WCAG criteria on an individual basis. If it’s very difficult to test a particular standard manually, use an automated tool to get some answers quickly. This should give you insight into little bits of code that you’ve missed or can’t see.

To get yourself off to the best start, it’s important to give yourself a good grounding in web development. Understand how websites are built, how to build code, and what semantics look like. 

Learn how to use a screen reader properly. This will reveal so many faults in a page that aren’t apparent to some testing tools, or to you as a sighted user. 

Familiarise yourself with the accessibility testing guidelines. These are the foundational elements of accessibility testing.

Pro tip: check out my article on accessibility audits for tips on how to test against WCAG criteria, prioritise your results, and action your findings.

It’s crucial to understand accessibility from a tech perspective, a user perspective, and a guidelines perspective.

Grant Broome

The future of accessibility testing tools

Every organisation that’s involved in testing or development should be looking at AI to assist them.

Why is AI so useful? Because while it’s more thorough and accurate, manual testing is very time consuming. The concept of getting a machine to accessibility decisions for you, thousands of times a second, is a pretty attractive proposition.

But the future and value of accessibility AI hugely depends on what we put into it. 

For AI to improve human decision making on accessibility, we need a grassroots, bottom up approach. Teams and organisations need to drive results together.

Olena Bulygina

Getting AI to the point where it improves our lives as accessibility testers will take time and effort. Teams and individuals will need to test hundreds of websites, pool our data, and teach AI what’s good and bad accessibility through repetition and feedback.

People like us need to help each other, and buy into the idea that the more data we have, the more useful the test results can be.

Grant Broome

Prime your accessibility testing for success

Compliance is a great place to start your accessibility journey – and automated testing tools provide a great way to quickly build a picture of what you need to fix to ensure your digital product is compliant.

But it’s important to remember that these tools have big limitations – and they’ll never be able to tell you if your website fully complies with WCAG.

To be sure that your site is accessible and meets WCAG standards, you need real people with specialist accessibility knowledge.

To ensure your digital product or service is compliant, you’ll need a hybrid approach that includes automated and human testing. Speak to our accessibility consultants today about how we can accelerate your journey to compliance.

You might also like...