Accessibility Testing

In today’s world where digital services are widespread, ensuring that software and web-based applications are accessible to everyone has become a fundamental requirement. Accessibility is not merely a technical standard; it is also a concept directly linked to social inclusion, equality, and human rights. Within this context, accessibility testing is a systematic type of software testing conducted to evaluate the usability of software products by individuals with various disabilities. The purpose of accessibility tests is to ensure that software is usable not only in terms of visual and auditory aspects but also with regard to cognitive, motor, and language impairments.

Conceptual Foundations and Scope of Accessibility

Accessibility

Accessibility refers to the effective usability of information and communication technologies (ICT) by individuals regardless of their physical, sensory, cognitive, or neurological differences. This concept is not only a matter of technical compliance but also a requirement for social participation and an integral part of user-centered design. The W3C (World Wide Web Consortium) defines accessibility as making websites, applications, and digital content perceivable, operable, understandable, and robust for people with disabilities.

Accessibility and Types of Disabilities

Accessibility is not limited solely to physical impairments such as vision or hearing loss. It encompasses a broad spectrum of functional disability types:

• Visual Disabilities (Vis): Color blindness, low vision, or blindness.
• Hearing Disabilities (Aud): Varying degrees from mild hearing loss to total deafness.
• Physical/Motor Disabilities (Phy): Reduced muscle control, coordination disorders, tremors, or paralysis.
• Speech Disabilities (Spe): Difficulty in producing intelligible speech.
• Cognitive Disabilities:
• • Attention and Memory (A&M): Difficulty following long sequences of tasks or retaining short-term information.
  • Higher-Level Reasoning (HLL): Challenges with abstraction, planning, and decision-making.
  • Language and Numeracy (L&N): Difficulty comprehending and processing dense text or numerical content.

This classification is an enhanced functional version of the traditional five-category system developed by W3C (visual, auditory, physical, cognitive/learning/neurological, and speech). Bai and colleagues^【1】 This structure promotes a more comprehensive evaluation by preventing accessibility testing from focusing exclusively on visual or physical impairments.

Dimensions of Digital Accessibility

Accessibility encompasses not only access but also interaction and comprehension. Within this framework, accessibility must be evaluated across four fundamental dimensions:

• Perceivable: Information must be detectable by users (e.g., providing alternative text for visual content).
• Operable: Users must be able to control the interface through various means such as keyboard, voice commands, or touch.
• Understandable: Interfaces and content must be intuitive, simple, and user-friendly.
• Robust: The system must function reliably across different technological environments and assistive technologies (e.g., screen reader compatibility).

These principles are defined by WCAG (Web Content Accessibility Guidelines) and apply not only to web platforms but also to mobile and desktop applications.

Accessibility and the Software Development Process

Accessibility is not a feature that can be added to software after development through testing; it must be integrated into the design process from the outset. Unfortunately, studies in the literature reveal that a significant proportion of software developers are either insufficiently familiar with accessibility guidelines or face challenges in implementing them.

Therefore, accessibility should be viewed as an integral component of usability, user experience (UX), and quality assurance (QA) processes. The growing adoption of automated testing tools and the evolution of accessibility into a continuously testable attribute play a key role in this transformation.

Classification of Accessibility Testing Methods

Accessibility testing employs various methods and tools to evaluate the usability of digital systems by individuals with disabilities. These methods can be categorized based on scope, accuracy, cost, ease of application, and their effectiveness across different disability types. Research in the literature demonstrates that accessibility testing cannot be limited to a single method; the best results are achieved through the combined use of multiple complementary approaches.

Automated Testing Tools (Auto)

Automated testing tools are programs that detect accessibility errors in software systems without human intervention. They typically identify visual and structural deficiencies such as improper HTML tagging, insufficient color contrast, or missing content descriptions. These tools are frequently preferred due to their low cost and speed. However, they are inadequate for semantic evaluation or addressing issues related to cognitive disabilities.

Notable tools:

• Android Accessibility Scanner: Detects issues such as missing content descriptions, low contrast, and small touch targets in Android applications.
• Wave, aXe, HTML CodeSniffer: Automatically scan web-based accessibility issues; provide limited but effective checks for visual and physical impairments.

Checklists and Standards-Based Audits (Check)

This method involves conducting accessibility checks based on international standards such as WCAG (Web Content Accessibility Guidelines). The tester manually evaluates each component of the system against a predefined checklist.

Advantages:

• Ensures compliance with legal and regulatory requirements.
• Can be documented and audited in written form.

Limitations:

• Time-consuming.
• Prone to errors depending on the tester’s knowledge and experience.

Simulation Tools (Sim)

In simulation-based testing, the impact of specific disability types on user experience is mimicked. For example, conditions such as low vision, color blindness, or dyslexia are simulated to test how accessible the interface is under these conditions.

Examples:

• Cambridge Simulation Glasses (color blindness visualization)
• WebAIM Low Vision Simulator, Funkify (digital simulation of various disability types)

These methods are effective in fostering empathy and helping designers visualize accessibility needs; however, they have limited capacity to generate quantitative data.

Assistive Technology-Based Testing (AT – Assistive Technology)

Assistive technologies are tools used by individuals with disabilities (e.g., screen readers, alternative keyboards). Testing through these technologies evaluates how well a software application works with them.

Common tools:

• Screen Readers: NVDA (Windows), VoiceOver (iOS), TalkBack (Android)
• Switch Access: Alternative interaction methods designed for users with physical interaction difficulties.

These tests are valuable because they closely reflect real user scenarios, but they require time and expertise to implement.

Expert-Based Methods (Exp)

In these methods, accessibility experts evaluate software by creating specific scenarios or user profiles (personas). Techniques such as heuristic evaluation, cognitive walkthrough, and barrier walkthrough fall within this category.

Characteristics:

• The Barrier Walkthrough method systematically evaluates barriers users may encounter.
• Persona Testing tests software using scenarios representing the characteristic traits of users with different types of disabilities.

While these methods enable in-depth evaluation of accessibility, they require high expertise and significant time investment.

Coverage of Methods by Disability Type

Accessibility testing methods are compared below according to the following disability types:

This table demonstrates that automated tools are inadequate for testing cognitive, numerical, and linguistic disabilities, and that this gap can only be addressed through expert-based or hybrid methods.

Challenges in Accessibility Testing

While accessibility testing aims to ensure that software systems are usable by individuals with disabilities, achieving this goal faces numerous technical, organizational, and methodological challenges. Findings indicate that accessibility is frequently neglected as a quality attribute throughout the software development lifecycle and that current testing practices contain significant shortcomings.

Scope Limitations and Inadequate Focus on Disability Types

The majority of current testing tools and methods focus primarily on visual and physical disabilities, while evaluations for cognitive, linguistic, auditory, and speech-based impairments remain insufficient. In particular, cognitive barriers such as attention and memory difficulties, impaired higher-level reasoning, or challenges in language and numeracy comprehension are inadequately represented in testing.

Limitations of Automated Tools

Although automated testing tools offer low-cost and rapid results, they suffer from limitations such as inability to perceive semantic context, interpret user intent, and rely solely on superficial rules.

For example:

• If multiple buttons are labeled “Remove,” an automated tool may detect the duplication but cannot determine which content each label refers to.
• Even if a content description exists, the tool cannot assess whether the description is meaningful or functionally relevant.

High Cost and Time Requirements of Testing

Expert-based accessibility testing, especially for cognitive disability assessments, is indispensable. However, these tests:

• Require intensive human resources,
• Can take hours per application,
• Necessitate specialized training for evaluators.

Additionally, organizing manual user testing—including recruiting participants with diverse disabilities and systematically analyzing results—represents a significant cost factor.

Difficulty Integrating into the Software Development Process

In many software projects, accessibility testing:

• Is deferred until the end of the project,
• Is viewed as an unnecessary expense,
• Is not included in test scenarios.

Accessibility checks must be integrated from the beginning of the software development process, yet many developers are either unfamiliar with accessibility guidelines or lack sufficient support to implement them. Particularly in mobile applications, dynamic user interfaces, device diversity, and platform-specific interactions further complicate this process.

Inadequacy of Assistive Technologies and Simulations

The use of simulations and assistive technologies is important for approximating real user experiences. However:

• Most of these tools simulate only visual or motor impairments.
• Effective simulation technologies for impairments such as speech, attention deficits, or decision-making difficulties are not widely available.
• Evaluations are often based on hypothetical scenarios rather than real user experiences.

Moreover, assistive technologies do not always function according to standards—for example, screen readers may fail to recognize certain custom components, thereby reducing the reliability of tests.

Interpretation of Test Outputs and Integration with Developers

Equally important as detecting accessibility errors is reporting them in a way that is understandable to developers. However, many automated tools provide insufficient user-friendly error descriptions and improvement suggestions. The technical knowledge required to fix these errors often creates implementation challenges, especially for inexperienced teams.

Test Scope by Functional Disability Type

The effectiveness of accessibility testing is directly related not only to which tools or methods are used but also to the extent to which these tools can cover specific disability types. While the W3C classification is commonly used to evaluate accessibility testing methods, an expanded, more detailed, and functionally oriented “disability categories” system has been proposed.

This system draws attention not only to physical impairments such as vision or hearing loss but also to limitations based on cognitive, linguistic, and higher-level mental processes. This enables deeper test coverage and clearly defines which methods address which disabilities.

Categorization of Functional Disability Types

The following table presents the proposed functional disability types and brief definitions:

This classification, unlike the traditional W3C structure, breaks down cognitive disabilities into three subcategories, enabling a more precise analysis in accessibility assessments.

Observations and Evaluation

• Best coverage is achieved through expert evaluation methods. However, due to their high cost and expertise requirements, these methods are rarely used in projects.
• Automated testing tools provide effective checks for visual and physical disabilities but are largely inadequate for complex impairments such as attention, memory, higher-level cognitive processes, and speech.
• Checklist-based audits offer the most balanced approach, as they can be configured to align with a broad range of disabilities. However, they remain dependent on the auditor’s knowledge.
• Simulation tools are valuable for building empathy but have limited practical test validity and are mostly restricted to visual impairments.
• Assistive technologies are successful in testing visual accessibility through tools such as screen readers. However, it must be remembered that these tools represent only specific user profiles and do not always ensure semantic adequacy.

Best Practices in Accessibility Testing

Improving the effectiveness of accessibility testing requires more than selecting tools or methods; the timing and manner of their application are equally critical. Empirical studies and industry guidelines highlight the need for strategic practices to ensure sustainable and inclusive accessibility testing.

Below are best practices for enhancing accessibility testing performance, organized under systematic headings.

Integrate Testing into the Software Development Lifecycle

Accessibility is often overlooked in software development processes or tested only at the end of product delivery. However, the best practice is to integrate accessibility principles throughout all stages of the software development lifecycle—analysis, design, development, testing, and maintenance.

• In the initial analysis phase: The needs of users with disabilities within the target audience must be defined.
• In the design phase: Mockups and user flows based on accessibility standards (e.g., WCAG) must be developed.
• In the development phase: Developers must pay attention to criteria such as content descriptions, keyboard access, and contrast ratios.
• In the testing phase: Both automated and manual accessibility tests must be applied, and findings must be integrated into the bug tracking system.

This approach transforms accessibility from a feature added after development into a fundamental component of product quality.

Develop a Hybrid Testing Strategy

Due to the multidimensional nature of accessibility, a single testing method is insufficient. Therefore, the best practice is to use complementary methods together—a strategy known as “test method triangulation.”

Recommended hybrid structure:

• Automated tools (e.g., Android Accessibility Scanner, aXe): Provide fast and repeatable scans.
• Checklists (e.g., WCAG, WebAIM): Offer manual compliance verification against standards.
• User testing: Seesional issues are identified by observing real users interacting with the system.
• Expert evaluation: Complex or semantic-level problems are assessed by experts.

This multi-layered structure ensures that accessibility testing is deep, not superficial, and sustainable.

Select Methods Based on Disability Type

Not every testing method covers all disability types equally. Therefore, when defining the test strategy, tools and methods must be selected according to the target user profiles and functional disability types.

For example:

• For visual disability-focused tests: Screen reader simulations and contrast analysis tools are appropriate.
• For cognitive disabilities: Expert-based persona tests and analyses of information density and task complexity are required.
• For physical disabilities: Minimum touch target tests or keyboard navigation tests are prioritized.

Establish a Repeatable and Continuous Testing Environment

Accessibility testing is not a one-time audit but a quality assurance step that must be integrated into continuous integration (CI) processes.

• Automated testing tools must be incorporated into CI/CD pipelines at regular intervals.
• Detected errors must be labeled as “critical” or “improvement suggestion,” and resolution timelines must be tracked.

This approach makes accessibility sustainable and reduces the risk of neglect.

Invest in Developer and Designer Education

Research shows that the majority of developers and design teams are insufficiently familiar with accessibility guidelines and face challenges in implementation.

To overcome this issue:

• Internal accessibility training programs must be organized,
• Practical content on WCAG, ARIA, and Android/iOS accessibility APIs must be provided,
• Accessibility criteria must be included in code reviews.

Education transforms accessibility from a “post-hoc fix” into a “preventive design factor.”

Meaningful and Actionable Reporting

Accessibility testing must not only detect errors but also generate actionable information for developers. To this end:

• Errors must be clearly explained with examples and technical accuracy.
• Recommended corrective steps must be provided for each error.
• Prioritization must be applied (e.g., “critical,” “important,” “warning”).
• Feedback loops must be kept short; communication between design, development, and testing teams must be strong.