LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection

2026-08-10

AI Summary

This work addresses the significant performance degradation of existing GUI grounding methods on small targets, densely packed widgets, and out-of-distribution interfaces, as well as their lack of mechanisms to reflect upon and refine predicted coordinates. The authors propose LookAgain, which reframes grounding as a closed-loop “predict–re-examine–refine” process: an initial localization module generates coordinate hypotheses with visual markers and crops corresponding local image regions, while a verification module leverages visual feedback to decide whether to accept or reject each hypothesis, enabling iterative refinement. Notably, this is the first approach to incorporate a post-prediction visual reflection mechanism that uses prior predictions as spatial priors for correction—rather than merely narrowing the observation window—establishing a reject-capable, closed-loop grounding paradigm. The model is initialized via supervised fine-tuning of reflection trajectories and further optimized through GRPO reinforcement learning using only final grounding accuracy as the reward signal. Experiments demonstrate that LookAgain achieves state-of-the-art performance across multiple GUI benchmarks, substantially outperforming existing methods particularly in scenarios requiring rejection capability and generalization.

Abstract

Recent graphical user interface (GUI) grounders have significantly advanced single-shot accuracy on standard benchmarks, yet their performance degrades sharply on small targets, densely packed controls and out-of-distribution interfaces. We attribute this gap to a paradigmatic limitation shared by existing approaches: none of them treats a produced coordinate as a hypothesis to be reflected upon and revised under new visual evidence. This manifests as three coupled issues: 1) Lack of post-hoc reflection. The prediction is frozen at the moment of emission, leaving no internal mechanism to challenge or refine it. 2) Visual evidence decoupled from the prediction. The auxiliary visual evidence is gathered to support the upcoming coordinate rather than to scrutinise the one already committed to. 3) Refinement over views, not over predictions. The iterative zoom-in refines the inspected region instead of inheriting a previous coordinate as a spatial prior to be corrected. In this paper, we propose LookAgain, a closed-loop GUI grounder driven by post-prediction visual reflection. LookAgain reformulates grounding as a multi-turn predict-look-again-refine process with two primitives: "locate" posts a coordinate hypothesis, renders a marker on the image and appends a local patch of the predicted region. It anchors the next reasoning step to the previous prediction as a spatial prior; "confirm" accepts or reject the hypothesis and terminates the procedure. We train the LookAgain grounder with SFT on constructed reflective trajectories as a cold start, followed by GRPO with terminal grounding correctness as the sole reward. Extensive experiments show that LookAgain consistently improves performance on both refusal-aware and general GUI grounding benchmarks, achieving state-of-the-art results. Comprehensive ablations further verify the effectiveness of the proposed framework.

Problem

• GUI grounding
• visual reflection
• closed-loop
• coordinate refinement
• out-of-distribution

Innovation

• closed-loop grounding
• visual reflection
• predict-look-again-refine
• spatial prior
• GUI grounder