RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection

2026-09-16

AI Summary

为解决GUI定位中准确性和效率之间的权衡问题，提出RankGround框架，通过轻量级重排器指导裁剪选择，实现单次视觉-语言模型调用下的高效精准定位。

Abstract

Graphical User Interface (GUI) grounding is a fundamental perception task for multimodal agents, enabling them to interpret natural language instructions and interact with digital interfaces. Existing methods face a fundamental trade-off between accuracy and efficiency: direct full-image inference often fails to capture small or visually similar UI elements, while multi-crop strategies improve localization at the cost of multiple expensive Vision-Language Model (VLM) calls per query. To address this challenge, we propose RankGround, a two-stage framework that achieves accurate GUI grounding with a single VLM call per query. Central to our approach is GroundRanker, a lightweight multimodal reranker that identifies the most promising crop from a dense candidate set. Because no off-the-shelf ranking dataset is available, we construct ranking supervision data from existing grounding datasets. A strict containment criterion and boundary-aware positive augmentation improve alignment and spatial coverage in cluttered layouts. GroundRanker is then trained with a two-stage curriculum: a pointwise objective first learns coarse containment, and a listwise objective refines subtle semantic and spatial distinctions among visually similar crops. Experimental results show that RankGround consistently outperforms strong baselines while reducing computational cost. It achieves 1.4 times faster inference and improves localization accuracy by 5.5% on average over the second-best method across all backbones and screen scales, establishing a new state of the art in both efficiency and precision for GUI grounding.

Problem

• Graphical User Interface
• Grounding
• Efficiency
• Accuracy
• Vision-Language Model

Innovation

• RankGround
• GroundRanker
• lightweight reranker
• single VLM call
• two-stage curriculum