Notes from "Towards Automated Accessibility Report Generation for Mobile Apps"
One of the most commonly used tools for generating audits on Apple platforms is the Accessibility Inspector, specially its audit functionality. The fact that this tool does not work for visionOS, in addition to many of the other reasons highlighted in this paper, decreased my use of it to nearly zero. But, there is still this feeling of void, a need for a kind of tool or a pipeline that provides for a better overview of the accessibility implementation on your app while also being more proactive with reduced friction on iterations. Currently, testing accessibility is such an extensive task given the wide range of accessibility settings available together with all the possible combinations of systems and devices, that it is practically impossible to test on every release, leaving the door open to regressions or, worse, neglecting the step altogether.
This is a where this type of research is welcome; the proposal involves using a novel mix of already existing tools powered by more efficiently trained machine learning models fine-tuned to address present limitations.
Quotes from the abstract:
developed a system to generate whole app accessibility reports by combining varied data collection methods (e.g., app crawling, manual recording) with an existing accessibility scanner
developed a screen grouping model with 96.9% accuracy (88.8% F1-score) and UI element matching heuristics with 97% accuracy (98.2% F1-score).
combined these technologies in a system to report and summarize unique issues across an app, and enable a unique pixel-based ignore feature to help engineers and testers better manage reported issues across their app’s lifetime
Main limitations of current approaches
Common implementations rely on app crawlers that randomly or through record & replay approaches crawl an app to detect accessibility issues
- They rely on accessible view hierarchies to drive the crawling itself which prior work has demonstrated to be often incomplete or unavailable for highly inaccessible apps
- None of these works has yet studied how users interact with an interpret information from these accessibility reports, and what features are important in an accessibility report generation tool.
Identified user needs for an accessibility report generation system
- Reduce the time required for developers to manually scan individual screens with accessibility auditing tools.
- Provide developers with an overall app accessibility report.
- Enable developers to reduce noise by ignoring false positive or previously addressed issues.
Observations
- While the participants sometimes wrote accessibility tests and used automated scanners, they reported primarily manually testing their apps
- Current tools provide no results overview
- Current tools are too noisy
- Manual scanning introduces inefficiencies
- Accessibility regressions can be created and persist for quite some time
- The reports contained many dynamic type issues, missing labels, poor contrast, and small target size, but some issues participants found are not detectable by our system and participants found them through manual testing.
- Participants may benefit from having a mode that supports both automated exploration and control over which user scenarios and tasks are explored by the app crawler.
- Supporting triaging and marking issues as ignored over time was also noted as an important feature in long term use, as participants mentioned they might be likely to file bugs or find issues in the report that they would mark as “won’t fix” or “minor issues”. These issues should then be filtered out of future reports automatically.
- One participant in the study is a screen reader user noted that for very inaccessible apps, our system would let them generate a more complete report compared to manual scanning tools that require use of the VoiceOver screen reader to navigate the app to each screen for auditing. Unexposed elements in very inaccessible apps might cause them to miss auditing key areas of the apps they would be unable to navigate.
This type of research highlights how artificial intelligence approaches can be used to generate solutions to previously unsolvable problems, and seeds the next generation of accessible tools.
Jeffrey P. Bigham