Problem
Research questions and friction points this paper is trying to address.
UI screen retrieval
feature-oriented similarity
design variation
UI clustering
abstraction gap
Fusing UI Structure & Semantics for Feature-Oriented App Screen Retrieval & Clustering
🤖 AI Summary
This work addresses the challenge of measuring functional similarity among application interfaces hindered by divergent design choices. To this end, we propose FRAME, a novel method that constructs symbolic graph representations to uniformly encode visual, semantic, and layout information of UI components. FRAME uniquely integrates neuro-symbolic systems with UI-specific computational geometry algorithms to enable weighted embedding propagation, effectively fusing multimodal features for modeling function-oriented screen similarity. Experimental results demonstrate that FRAME significantly outperforms strong baselines across three benchmarks, achieving up to a 13% relative improvement in Mean Reciprocal Rank (MRR) on retrieval tasks and a 7.6 percentage point gain in clustering accuracy.
📝 Abstract
User Interface (UI) programming is challenging due to the complex abstraction gap between code and graphical software representations. To bridge this gap, UI programming tools often rely on screen retrieval and clustering, which require accurate similarity measures based on overlapping features. However, computing feature-oriented similarity is difficult because screens with similar functionality often exhibit design variations.
To address this, we propose FRAME (ReinForced UseR InterfAce Screen EMbedding with Graphical Structural ComprEhension), a multi-modal, neuro-symbolic embedding technique. FRAME constructs symbolic, graph-based representations of UI components to encode salient relationships and capture feature patterns across different screens. It leverages large vision-language models for visual and lexical encoding, alongside a novel UI-specific computational geometry algorithm that enables weighted embedding propagation. Across three benchmarks, FRAME outperforms strong baselines by up to 13% MRR in search and 7.6 percentage points in clustering accuracy. A comprehensive ablation study further confirms the benefit of each component, demonstrating FRAME's potential for enhancing automated UI design and testing tools.
Innovation
Methods, ideas, or system contributions that make the work stand out.
neuro-symbolic embedding
UI screen retrieval
graph-based representation
vision-language models
computational geometry
🔎 Similar Papers
No similar papers found.
