QF-Test Beta-Test Entwicklerversion mit a11y-Checks

Accessibility is not an optional aspect of modern software development, especially for public sector applications that must comply with the requirements of the BITV 2.0 and the Accessibility Strengthening Act (BFSG). Automated testing plays an important role for digital products. In this blog post, I compare the tools QF-Test, Cypress Axe and axe-core CLI in terms of their accessibility testing capabilities.

Personal experience with QF-Test

At this point, I would like to report on my personal experiences with the developer version of QF-Test, which was provided to me by the manufacturer QFS as a pre-release version. Even this version was surprisingly stable and the customer support responded quickly and reliably to a problem that occurred on my own computer.

The final version 9.0.0 will be released in December 2024 and can be used with test licences immediately after release.

QF-Test: Comprehensive solution for accessibility testing

QF-Test is a tool for test automation of GUI applications and offers advanced features for accessibility testing.

The advantages of QF-Test

• Libraries are always up to date: QF-Test regularly integrates the latest accessibility libraries and thus always remains at the cutting edge of technology.
• Extended tests: In addition to standard tests, QF-Test offers additional checks, such as colour contrast checks in images, to ensure that embedded graphics are also accessible.
• Modular approach: QF-Test also supports model-based testing and the direct generation of test data, thanks to additional modules, which increases efficiency in complex projects.
• Improved reporting: The tool creates detailed reports with screenshots of the affected components, which facilitates error analysis and communication within the team.

Cypress Axe: Integration in end-to-end tests

Cypress Axe is based on the axe-core library and allows the integration of accessibility tests into end-to-end tests with Cypress.

The advantages of Cypress Axe:

• Seamless integration: Cypress Axe can be integrated directly into existing Cypress test environments, allowing accessibility checks to be performed in parallel with other tests.

The limitations of Cypress Axe:

• Outdated libraries: The last update of the GitHub repository was more than two years ago, which indicates a lack of maintenance.
• Limited checks: Compared to QF-Test, advanced features such as colour contrast checks in images are missing.
• Limited reporting: Reporting is less detailed and does not offer visual documentation such as screenshots.

axe-core CLI: The framework-independent solution

axe-core CLI is a flexible tool for accessibility testing that can be used independently of specific frameworks.

The advantages of axe-core CLI:

• Framework independence: The CLI can be combined with various tools and technologies, offering a high degree of flexibility.
• Cross-browser testing: It supports various browsers and is therefore suitable for tests in CI/CD pipelines.
• Easy integration: It can be easily integrated into scripts and pipelines.

The limitations of axe-core CLI

• Less convenient: Compared to integrations such as Cypress Axe, specific DOM interactions are missing.
• No visual reporting: The results are displayed in text form, without visual aids such as screenshots.
• Missing advanced checks: Functions such as colour contrast checks in images are not available.

Comparison of tools

The tool best suited for accessibility testing depends on the various project requirements. QF-Test offers extensive functionality and regular updates, making it particularly suitable for complex projects. Cypress Axe enables easy integration into existing Cypress environments, but has limitations in terms of up-to-dateness and functionality. The axe-core CLI offers flexibility and is suitable for use in CI/CD pipelines, but has fewer convenience features and visual reporting.

Accessibility is an ongoing process that requires continuous attention. Choosing the right tool can be crucial to making digital products accessible to all users and meeting legal requirements. On 5 October, WCAG version 2.2 was presented, which contains nine additional criteria.

Further advantages of QF-Test

QF-Test offers numerous advantages that make it a versatile tool for accessibility testing and GUI test automation. In addition to the features already mentioned, such as the modular approach, the test data generator and the extensive reporting, QF-Test also offers a number of other impressive functions:

Technology support

QF-Test supports several well-known technologies, including Java Swing, JavaFX, web and Windows applications. It is therefore not limited to web applications and offers flexibility in heterogeneous software landscapes.

• 1. Model-based approach: Tests can be based on reusable modules, which significantly simplifies test maintenance and scaling. This approach saves teams a considerable amount of time, especially when it comes to large applications.
• 2. Test data generator: QF-Test can be used to dynamically create and integrate test data into tests. This is particularly useful for scenarios in which different user contexts or input data need to be tested.
• 3. Easy integration into CI/CD pipelines: QF-Test can be seamlessly integrated into common CI/CD environments such as Jenkins, GitLab CI or Azure DevOps. This enables regular and automated accessibility testing throughout the entire development process. It also allows reporting results to be returned to test management systems such as Zephyr Scale and XRay for Jira. This provides a better overview of the presentation of automated and manual tests.
• 4. Advanced reporting features: QF-Test's reporting goes beyond simple log output. With detailed reports, screenshots of the affected components and clearly visualised errors, troubleshooting is accelerated.
• 5. Platform independence: QF-Test is available for all major operating systems such as Windows, macOS and Linux. This makes it easier for teams working in different environments to collaborate.
• 6. Experience and stability: QF-Test is an established tool with years of development and an experienced team behind it. Customer support is fast and reliable, which has been confirmed in my personal experience.

Iterative approach

Depending on the technical situation and resources available for the project, it may also make sense to start with axe core or Cypress Axe for automated accessibility testing and switch to QF-Test at a later stage. The manufacturer also offers a test phase for this purpose, during which the advantages for your own projects can be evaluated. The licence model, which is based on lifelong floating licences, is also interesting. This means that QF-Test can also be used for multiple projects and allows you to react flexibly to changes in team members.

Conclusion

With its extensive range of features, QF-Test offers a solid and scalable solution for accessibility testing and automating GUI tests. The combination of technological breadth, functional depth and user-friendly integration makes it an excellent choice for projects that require more than just standard accessibility testing.

For more information on using Cypress Axe and axe core via CLI, see these two blog posts:

• Automated accessibility testing with cypress-axeecks
• Automated accessibility testing with cypress-axe – Part 2