Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding

2026-09-14

AI Summary

为解决GUI定位中高昂标注成本问题，提出基于置信度锚定的负样本学习方法CANL，利用坐标令牌置信度筛选伪标签并优化模型。

Abstract

Graphical User Interface (GUI) grounding is essential for autonomous agents to map natural language instructions to precise screen coordinates. However, existing supervised fine-tuning and reinforcement learning methods are constrained by the high cost of annotation, creating a scalability bottleneck. In this paper, we introduce a label-free test-time training paradigm driven by two key insights: (1) confidence patterns in coordinate tokens are a better indicator than full-sequence confidence, and (2) in sparse GUI coordinate spaces, negative samples offer more reliable learning signals than potentially noisy positive ones. We first propose Confidence-Anchored Learning (CAL), which utilizes coordinate-token confidence to filter pseudo-labels and assign distance-based binary rewards. Building on this, we develop Confidence-Anchored Negative Learning (CANL), which exclusively optimizes the model using negative samples to bypass the risks of incorrect positive samples. Experimental results demonstrate that CANL-7B achieves 92.1% on ScreenSpot-V2. On more challenging ScreenSpot-Pro, CANL-7B reaches 33.8%, an 8.9% absolute improvement over the base model. Our findings establish coordinate-token confidence as a powerful alternative to manual annotations for scalable GUI agent development.

Problem

• Graphical User Interface (GUI) grounding
• natural language instructions
• annotation cost
• scalability bottleneck

Innovation

• Confidence-Anchored Learning
• Negative Samples
• Test-Time Adaptation
• Coordinate-Token Confidence
• Pseudo-Label Filtering