AXNav: Replaying Accessibility Tests from Natural Language
AuthorsMaryam Taeb, Amanda Swearngin, Eldon Schoop, Regina Cheng, Yue Jiang, Jeff Nichols
AuthorsMaryam Taeb, Amanda Swearngin, Eldon Schoop, Regina Cheng, Yue Jiang, Jeff Nichols
Despite increasing awareness of the need to support accessibility in mobile apps, many still lack support for key accessibility features. Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecyle. However, manual testing can be tedious, often has an overwhelming scope, and test passes can be difficult to time amongst other development milestones. Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs; however, none have yet explored their use in controlling assistive technologies for the purposes of supporting accessibility testing. In this paper, we explore the requirements of natural language based accessibility testing workflow through a formative study. Based on this, we present a system that takes as input a manual accessibility test (e.g., "Search for a show in VoiceOver") and uses an LLM combined with pixel-based UI Understanding models to convert the test into a chaptered, navigable video that a QA tester can use to pinpoint issues. In each video, we apply heuristics to detect and flag accessibility issues (e.g., Text size not increasing with Large Text enabled, VoiceOver navigation loops) to help QA testers more easily pinpoint issues. We evaluate this system through a 10 participant user study with accessibility QA professionals who indicated that the tool would be very useful in their current work and gave us several promising directions for future work.
1) Title: iOS: VoiceOver: Search for a Show
Go to Settings > Accessibility > VoiceOver, and enable VoiceOver (VO)
Search for a show and verify that everything works as expected and there are accurate labels
Turn off VO and verify that searching for a show works as expected
2) iOS: Podcasts: Dynamic Text in Search Tab
In Settings > Accessibility > Display & Text Size, enable larger text and set to maximum size
Verify all text (titles, headers, etc.) font size has adjusted consistently
3) iOS: Podcasts: Button Shapes across app
Expected Result: When Testing button shapes- we want to make sure that all text (not emojis or glyphs) get underlined if they are NOT inside of a button shape already. If the text is already within a button shape, it is a bug!