Should you use accessibility testing tools?
Accessibility testing tools aim to help you improve the compliance of your site or app. But do they deliver on what they promise? And where do they fit in your accessibility testing toolkit?
There are tonnes of automated tools on the market, with wildly different claims. Some promise a passport to full compliance – others, merely some relief from your compliance headaches.
It’s a minefield. So to make sense of it all, I caught up with Dig Inclusion founder, Grant Broome, for CXcon: Rethink Accessibility – Inviqa’s online digital accessibility conference.
Together we explored the how, what, why, and when of accessibility testing tools and answered your questions live.
Keep reading to get the key takeaways from our conversation. Or – if you have a little more time – you can check out the below video for the full talk.
What are automated accessibility testing tools?
Accessibility testing tools provide an automated way to check how well your website complies with the web content accessibility guidelines (WCAG).
They scan your website pages and check them against WCAG criteria, giving you a score or grade that helps you gauge your level of compliance. They also help you identify what you need to improve to make your digital product more compliant.
Google Lighthouse is an example of an automated accessibility testing tool. It provides a clear gauge of digital accessibility, using a tangible score that you can track and measure yourself against.
The benefits of accessibility testing tools
Automated testing tools can be a great starting point for understanding how well your digital prototype or product complies with the web content accessibility guidelines.
They’re especially handy if you need to scan a large volume of site pages and discover compliance issues quickly.
Let’s say you’re looking for missing image alt text. An automated tool can extract and serve you this information very quickly, while a manual check would take you ages.
Another great use case is checking forms. An automated tool can very quickly tell you whether your forms are missing labels, or are labelled twice. It’s a lot easier than manually checking your visual labels, and the ones lurking in your code and containers.
Automated testing improves and streamlines your manual testing – and it should be part of every accessibility tester's toolkit.
Accessibility testing tool examples
These are some of the automated tools I recommend and use daily in my accessibility work.
- WAVE Evaluation Tool. This is a great browser plugin from WebAIM designed to support your manual evaluation and help you better understand accessibility issues.
- Google Lighthouse. This is a great feel tool that the whole team can use. You’ll need to go through the accessibility evaluations manually, but the accessibility scoring is very valuable.
- Stark contrast checker. This contrast checker for Sketch helps you ensure that accessibility is baked into your digital product design.
The limitations of accessibility testing tools
Accessibility testing tools are super useful, but it’s important to understand their uses and limitations.
Many of these tools are oversold as a complete solution to compliance. But this just isn’t the case. And relying on these tools as your only solution for testing accessibility is hugely limiting.
Here are the cons or limitations you should know about.
Accessibility testing tools don’t guarantee compliance
Firstly, compliance does not guarantee accessibility.
It’s possible to build a web page with a perfect Lighthouse score that’s still not accessible. Only by testing with real users who experience disability can you understand whether your digital product is genuinely accessible.
The second key point here is that testing tools don’t guarantee compliance.
Many tools are unclear on how their algorithms test your site, which WCAG version they test against (for example, WCAG 2.1), or even which WCAG criteria they test against.
If you’re experienced in accessibility and testing, you can do your due diligence and build a picture of how well a tool serves your needs. But it’s important to ensure you’re not relying on these tools alone to achieve compliance.
Testing tools can be unreliable
Different testing tools give you different results. You can pass a test with one tool and fail it using another.
If you don’t have a background in accessibility, it’s difficult to know whether the results you receive from a tool are reliable or not.
Decision making is limited
Testing tools augment our decision making around accessibility. But they don’t have the context and human experience that’s needed to make decisions. And accessibility testing involves a lot of considered decision making.
Take alt text. A tool can confirm the presence of image alt text. But it can’t say with any certainty whether the alternative text correctly describes the contents of an image. And it doesn’t understand context.
Yes, it can flag alt text that looks strange or might be incorrect. For example, where it includes a string of numbers. But it can’t actually decide for itself whether alt text is valid.
To illustrate the point, Inviqa partner and Dig Inclusion founder Grant Broome, tested the same image of a kitten using the Google accessibility tester.
First he checked the image with the alt text ‘kitten’. Then he tested the same image with the alt text ‘great white shark’.
Both versions scored 100% using the tool. The tool obviously doesn’t know its kittens from its sharks. But it also can’t assess the context of the image and deduce – for example – that the alt text describes a kitten that’s actually named ‘Great White Shark’.
It’s the perfect example of why automated testing requires human validation.
And there are other considerations. One tool rarely does everything, and you need to be as close as possible to the production code of your website.
How to use tools in your accessibility testing
So how do you navigate these factors and use tools to improve your accessibility testing?
Start by looking at WCAG criteria on an individual basis. If it’s very difficult to test a particular standard manually, use an automated tool to get some answers quickly. This should give you insight into little bits of code that you’ve missed or can’t see.
To get yourself off to the best start, it’s important to give yourself a good grounding in web development. Understand how websites are built, how to build code, and what semantics look like.
Learn how to use a screen reader properly. This will reveal so many faults in a page that aren’t apparent to some testing tools, or to you as a sighted user.
Familiarise yourself with the accessibility testing guidelines. These are the foundational elements of accessibility testing.
Pro tip: check out my article on accessibility audits for tips on how to test against WCAG criteria, prioritise your results, and action your findings.
It’s crucial to understand accessibility from a tech perspective, a user perspective, and a guidelines perspective.
The future of accessibility testing tools
Every organisation that’s involved in testing or development should be looking at AI to assist them.
Why is AI so useful? Because while it’s more thorough and accurate, manual testing is very time consuming. The concept of getting a machine to accessibility decisions for you, thousands of times a second, is a pretty attractive proposition.
But the future and value of accessibility AI hugely depends on what we put into it.
For AI to improve human decision making on accessibility, we need a grassroots, bottom up approach. Teams and organisations need to drive results together.
Getting AI to the point where it improves our lives as accessibility testers will take time and effort. Teams and individuals will need to test hundreds of websites, pool our data, and teach AI what’s good and bad accessibility through repetition and feedback.
People like us need to help each other, and buy into the idea that the more data we have, the more useful the test results can be.
Prime your accessibility testing for success
Compliance is a great place to start your accessibility journey – and automated testing tools provide a great way to quickly build a picture of what you need to fix to ensure your digital product is compliant.
But it’s important to remember that these tools have big limitations – and they’ll never be able to tell you if your website fully complies with WCAG.
To be sure that your site is accessible and meets WCAG standards, you need real people with specialist accessibility knowledge.
To ensure your digital product or service is compliant, you’ll need a hybrid approach that includes automated and human testing. Speak to our accessibility consultants today about how we can accelerate your journey to compliance.
You might also like...
- Learn how to run your own accessibility audit
- Your guide to accessible content
- Upskill your teams with remote accessibility training
