V2P: Visual Attention Calibration for GUI Grounding via Background Suppression and Center Peaking

2026-01-11

AI Summary

This work addresses the limitations of existing GUI element localization methods, which often neglect the uncertainty in spatial interactions and the hierarchical nature of visual semantics, while conventional attention mechanisms are prone to background interference and struggle to accurately distinguish between the center and periphery of target elements. To overcome these challenges, the authors propose V2P, a novel approach that integrates background-suppressed attention with Fitts’ law–inspired 2D Gaussian heatmap modeling. The former suppresses distractions from irrelevant regions, while the latter adaptively generates center-focused heatmaps based on target size, emulating human visual attention for more precise calibration. Experimental results demonstrate that V2P achieves localization accuracies of 92.4% and 52.5% on the ScreenSpot-v2 and ScreenSpot-Pro benchmarks, respectively, with ablation studies confirming the effectiveness and generalization capability of its components.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Local Search

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

Precise localization of GUI elements is crucial for the development of GUI agents. Traditional methods rely on bounding box or center-point regression, neglecting spatial interaction uncertainty and visual-semantic hierarchies. Recent methods incorporate attention mechanisms but still face two key issues: (1) ignoring processing background regions causes attention drift from the desired area, and (2) uniform modeling the target UI element fails to distinguish between its center and edges, leading to click imprecision. Inspired by how humans visually process and interact with GUI elements, we propose the Valley-to-Peak (V2P) method to address these issues. To mitigate background distractions, V2P introduces a suppression attention mechanism that minimizes the model's focus on irrelevant regions to highlight the intended region. For the issue of center-edge distinction, V2P applies a Fitts' Law-inspired approach by modeling GUI interactions as 2D Gaussian heatmaps where the weight gradually decreases from the center towards the edges. The weight distribution follows a Gaussian function, with the variance determined by the target's size. Consequently, V2P effectively isolates the target area and teaches the model to concentrate on the most essential point of the UI element. The model trained by V2P achieves the performance with 92.4\% and 52.5\% on two benchmarks ScreenSpot-v2 and ScreenSpot-Pro (see Fig.~\ref{fig:main_results_charts}). Ablations further confirm each component's contribution, underscoring V2P's generalizability in precise GUI grounding tasks and its potential for real-world deployment in future GUI agents.

Problem

• GUI grounding
• visual attention
• background suppression
• center-edge distinction
• click imprecision

Innovation

• Visual Attention Calibration
• Background Suppression
• Center Peaking
• Gaussian Heatmap
• GUI Grounding