AccessFixer: Enhancing GUI Accessibility for Low Vision Users With R-GCN Model

2024-02-01

IEEE Transactions on Software Engineering

AI Summary

Low-vision users often struggle to perceive mobile GUIs due to excessively small component sizes, insufficient inter-component spacing, and inadequate color contrast. To address this, we propose the first end-to-end collaborative GUI accessibility repair method. Our approach models spatial, semantic, and stylistic relationships among UI components using a Relational Graph Convolutional Network (R-GCN) and jointly optimizes component size, spacing, and color attributes under design-consistency constraints—ensuring uniform color palettes, equidistant layouts, and scalable adaptations—while preventing new accessibility defects. Evaluated on 30 real-world mobile applications, our method resolves 81.2% of identified accessibility issues on average. We contributed pull requests to 10 open-source projects, with 8 accepted or merged. A user study with low-vision participants confirms statistically significant improvements in interface usability and perceived satisfaction.

Technology Category

• Humans and AI: AI for Accessibility
• Computer Vision: Multi-modal Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The Graphical User Interface (GUI) plays a critical role in the interaction between users and mobile applications (apps), aiming at facilitating the operation process. However, due to the variety of functions and non-standardized design, GUIs might have many accessibility issues, like the size of components being too small or their intervals being narrow. These issues would hinder the operation of low vision users, preventing them from obtaining information accurately and conveniently. Although several technologies and methods have been proposed to address these issues, they are typically confined to issue identification, leaving the resolution in the hands of developers. Moreover, it can be challenging to ensure that the color, size, and interval of the fixed GUIs are appropriately compared to the original ones. In this work, we propose a novel approach named AccessFixer (Accessibility Issues Fixing Method), which utilizes the Relational-Graph Convolutional Neural Network (R-GCN) to simultaneously fix three kinds of accessibility issues, including small sizes, narrow intervals, and low color contrast in GUIs. With AccessFixer, the fixed GUIs would have a consistent color palette, uniform intervals, and adequate size changes achieved through coordinated adjustments to the attributes of related components. Our experiments demonstrate the effectiveness and usefulness of AccessFixer in fixing GUI accessibility issues. After fixing 30 real-world apps, our approach solves an average of 81.2% of their accessibility issues. Compared with the baseline tool that can only fix size-related issues, AccessFixer not only fixes both the interval and color contrast of components, but also ensures that no new issues arise in the fixed results. Also, we apply AccessFixer to 10 open-source apps by submitting the fixed results with pull requests (PRs) on GitHub. The results demonstrate that developers approve of our submitted fixed GUIs, with 8 PRs being merged or under fixing. A user study examines that low vision users host a positive attitude toward the GUIs fixed by our method.

Problem

• Improve GUI accessibility for low vision users
• Fix small sizes, narrow intervals, low color contrast
• Ensure consistent and uniform GUI adjustments

Innovation

• R-GCN model
• fixing GUI issues
• consistent color palette