Wingspan LabsStart a Conversation

Rapid systematic evidence review

Automated Accessibility Scores Are Not Accessibility Evidence

What automated web testing can detect, what it misses, and how organizations should interpret the results

By Kevin Hintzman

Read the HTML summaryDownload PDF

Tagged PDF, 12 pages, 580 KB.

Research organization, drafting, and layout prepared with AI assistance under the author's direction.

Cover of the Wingspan Labs research brief Automated Accessibility Scores Are Not Accessibility Evidence
Wingspan Labs Research Brief, July 2026.

HTML summary

A score answers a narrower question than it appears to answer.

Automated tools can identify real, code-detectable accessibility issues quickly and consistently. Their scores remain partial measurements of a particular tool, version, ruleset, configuration, page sample, and moment in time.

An automated score is evidence about an automated test. It is not, by itself, evidence that a website is accessible.

Automated Accessibility Scores Are Not Accessibility Evidence

What the review examined

A transparent synthesis without a manufactured universal percentage.

This is a rapid review by one evidence reviewer. It is transparent, but it is not exhaustive.

300
records returned by the OpenAlex and Crossref API searches
278
unique records after deduplication
20
primary studies included in the synthesis
1
secondary systematic mapping included

A statistical meta-analysis was considered and rejected because the studies measured different constructs with incompatible denominators and generally did not report poolable uncertainty data.

A practical evidence model

Three layers answer three different questions.

01

Automated checks

What repeatable, code-detectable findings did this tool identify?

02

Experienced manual review

What requires context, judgment, interaction, and a reproducible method?

03

Task-based user evidence

What happens when disabled people complete meaningful tasks with their own strategies and technologies?

Before you rely on a score

Ask what the number can actually support.

The full paper explains each question and provides a ten-question checklist for vendors and internal teams.

  1. 01

    What exactly was scanned? A homepage cannot stand in for a form, document, account workflow, or state that was never tested.

  2. 02

    Which tool, version, rules, and configuration produced it? Without provenance, the result cannot be reproduced or compared meaningfully.

  3. 03

    What could the tool decide automatically? Separate definite failures, potential issues, and unresolved manual checks.

  4. 04

    Who resolved the judgment calls? Experienced review and a defined method matter.

  5. 05

    What interaction testing was completed? Keyboard, focus, zoom, reflow, screen-reader use, errors, touch, and dynamic states require interaction.

  6. 06

    What claim is the score being used to support? A bounded scanner result is not a conformance, accessibility, or legal determination.

Claims and limitations

Useful evidence still needs honest boundaries.

The review supports the conclusion that automated accessibility testing is useful but incomplete and must be interpreted in context. It does not establish the accessibility or WCAG conformance of Wingspan Labs, HorizonSong, or a client website; legal compliance; a universal automation coverage percentage; superiority of a scanner or methodology; customer outcomes; or market demand.

The search used broad scholarly APIs rather than subscription databases, one reviewer conducted screening and extraction, study methods varied substantially, and publication bias remains possible.

Cited sources

References

  1. Abduganiev, S. G. (2017). Towards automated web accessibility evaluation: A comparative study. International Journal of Information Technology and Computer Science, 9(9), 18-44. DOI 10.5815/ijitcs.2017.09.03
  2. Acosta-Vargas, P., Salvador-Ullauri, L. A., & Lujan-Mora, S. (2019). A heuristic method to evaluate web accessibility for users with low vision. IEEE Access, 7, 125634-125648. DOI 10.1109/ACCESS.2019.2939068
  3. Alsaeedi, A. (2020). Comparing web accessibility evaluation tools and evaluating the accessibility of webpages: Proposed frameworks. Information, 11(1), 40. DOI 10.3390/info11010040
  4. AlSaeed, D., Alkhalifa, H., Alotaibi, H., Alshalan, R., Al-Mutlaq, N., Alshalan, S., Bintaleb, H. T., & AlSahow, A. M. (2020). Accessibility evaluation of Saudi e-government systems for teachers: A visually impaired user's perspective. Applied Sciences, 10(21), 7528. DOI 10.3390/app10217528
  5. Ara, J., & Sik-Lanyi, C. (2025). Automated evaluation of accessibility issues of webpage content: Tool and evaluation. Scientific Reports, 15, 9516. DOI 10.1038/s41598-025-92192-5
  6. Brajnik, G. (2008). A comparative test of web accessibility evaluation methods. In Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 113-120). DOI 10.1145/1414471.1414494
  7. Brajnik, G., Yesilada, Y., & Harper, S. (2011). The expertise effect on web accessibility evaluation methods. Human-Computer Interaction, 26(3), 246-283. DOI 10.1080/07370024.2011.601670
  8. Centeno, V. L., Kloos, C. D., Fisteus, J. A., & Alvarez, L. A. (2006). Web accessibility evaluation tools: A survey and some improvements. Electronic Notes in Theoretical Computer Science, 157(2), 87-100. DOI 10.1016/j.entcs.2005.12.048
  9. Deeks, J. J., Higgins, J. P. T., Altman, D. G., McKenzie, J. E., & Veroniki, A. A. (2024). Chapter 10: Analysing data and undertaking meta-analyses. In J. P. T. Higgins et al. (Eds.), Cochrane Handbook for Systematic Reviews of Interventions (Version 6.5). Cochrane Handbook, Chapter 10
  10. Fischer, T., Lundell, B., & Gamalielsson, J. (2025). Coverage of web accessibility guidelines provided by automated checking tools. Universal Access in the Information Society, 24, 3615-3637. DOI 10.1007/s10209-025-01263-x
  11. Hartman, P., & Gorichanaz, T. (2025). Evaluating AI-powered website accessibility overlays. In Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 1-4). DOI 10.1145/3663547.3759761
  12. Ismailova, R., & Inal, Y. (2022). Comparison of online accessibility evaluation tools: An analysis of tool effectiveness. IEEE Access, 10, 58233-58239. DOI 10.1109/ACCESS.2022.3179375
  13. Kelsey-Adkins, E. R., & Thompson, R. H. (2022). Inter-rater reliability of command-line web accessibility evaluation tools. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility. DOI 10.1145/3517428.3550395
  14. Kumar, K. L., & Owston, R. (2016). Evaluating e-learning accessibility by automated and student-centered methods. Educational Technology Research and Development, 64(2), 263-283. DOI 10.1007/s11423-015-9413-6
  15. Kumar, S., Jeevitha Shree, D. V., & Biswas, P. (2021). Comparing ten WCAG tools for accessibility evaluation of websites. Technology and Disability, 33(3), 195-209. DOI 10.3233/TAD-210329
  16. Mateus, D. A., Silva, C. A., de Oliveira, A. F. B. A., Costa, H., & Freire, A. P. (2021). A systematic mapping of accessibility problems encountered on websites and mobile apps: A comparison between automated tests, manual inspections and user evaluations. Journal on Interactive Systems, 12(1), 145-171. DOI 10.5753/jis.2021.1778
  17. Palmer, Z. B., & Oswal, S. K. (2024). Constructing websites with generative AI tools: The accessibility of their workflows and products for users with disabilities. Journal of Business and Technical Communication. DOI 10.1177/10506519241280644
  18. Page, M. J., McKenzie, J. E., Bossuyt, P. M., et al. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. DOI 10.1136/bmj.n71
  19. Power, C., Freire, A., Petrie, H., & Swallow, D. (2012). Guidelines are only half of the story: Accessibility problems encountered by blind users on the web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 433-442). DOI 10.1145/2207676.2207736
  20. Tzimas, I., & Katsanos, C. (2021). Effect of potential issues flagged by automated tools on web accessibility evaluation results: A case study on university department websites. In Proceedings of the 25th Pan-Hellenic Conference on Informatics. DOI 10.1145/3503823.3503845
  21. Tzimas, I., & Katsanos, C. (2024). Do potential issues reported by automated tools affect guideline-based web accessibility evaluation? A case study on Greek public hospitals. In Proceedings of the 28th Pan-Hellenic Conference on Progress in Computing and Informatics. DOI 10.1145/3716554.3716596
  22. Vigo, M., Brown, J., & Conway, V. (2013). Benchmarking web accessibility evaluation tools: Measuring the harm of sole reliance on automated tests. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (pp. 1-10). DOI 10.1145/2461121.2461124
  23. Vollenwyder, B., Petralito, S., Iten, G. H., Bruhlmann, F., Opwis, K., & Mekler, E. D. (2023). How compliance with web accessibility standards shapes the experiences of users with and without disabilities. International Journal of Human-Computer Studies, 170, 102956. DOI 10.1016/j.ijhcs.2022.102956
  24. W3C Web Accessibility Initiative. (n.d.-a). Selecting web accessibility evaluation tools. Retrieved July 20, 2026, from W3C guidance on selecting evaluation tools
  25. W3C Web Accessibility Initiative. (n.d.-b). Template for accessibility evaluation reports. Retrieved July 20, 2026, from W3C accessibility evaluation report template

Complete publication

Read the full evidence review and references.

The PDF includes the complete method, synthesis, practical reporting guidance, claims boundaries, limitations, and cited sources.

Download the PDF
Author
Kevin Hintzman
Publisher
Wingspan LLC
Published
Document ID
AUTOMATED-ACCESSIBILITY-EVIDENCE-REVIEW-20260720-R1

Continue exploring

What makes an accessibility review meaningful? Explore Web Accessibility services

Next step

Need to understand what an accessibility score does—and does not—tell you?

Describe the site, user journey, and decision in non-sensitive terms. A scoped conversation can identify the evidence your organization actually needs.

Discuss an Accessibility Review