AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement

2026-03-18

AI Summary

This work addresses the challenge of grounding visual language models in high-resolution GUI screenshots, where fine-grained interface elements and ambiguous user instructions hinder accurate localization. To tackle this, the authors propose an adaptive scaling framework that integrates an instruction refinement module to rewrite vague commands into precise descriptions and employs a conditional scaling strategy to perform secondary inference on initially predicted small elements, balancing accuracy and efficiency. A high-quality GUI grounding dataset is curated, and Group Relative Policy Optimization (GRPO) is leveraged to jointly optimize click coordinates and bounding box predictions. Evaluated on public benchmarks, the method achieves state-of-the-art performance among comparable or larger models, significantly enhancing high-resolution GUI understanding and agent deployment effectiveness.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

GUI grounding is a critical capability for vision-language models (VLMs) that enables automated interaction with graphical user interfaces by locating target elements from natural language instructions. However, grounding on GUI screenshots remains challenging due to high-resolution images, small UI elements, and ambiguous user instructions. In this work, we propose AdaZoom-GUI, an adaptive zoom-based GUI grounding framework that improves both localization accuracy and instruction understanding. Our approach introduces an instruction refinement module that rewrites natural language commands into explicit and detailed descriptions, allowing the grounding model to focus on precise element localization. In addition, we design a conditional zoom-in strategy that selectively performs a second-stage inference on predicted small elements, improving localization accuracy while avoiding unnecessary computation and context loss on simpler cases. To support this framework, we construct a high-quality GUI grounding dataset and train the grounding model using Group Relative Policy Optimization (GRPO), enabling the model to predict both click coordinates and element bounding boxes. Experiments on public benchmarks demonstrate that our method achieves state-of-the-art performance among models with comparable or even larger parameter sizes, highlighting its effectiveness for high-resolution GUI understanding and practical GUI agent deployment.

Problem

• GUI grounding
• vision-language models
• high-resolution images
• ambiguous instructions
• small UI elements

Innovation

• instruction refinement
• adaptive zoom
• conditional zoom-in
• GUI grounding
• GRPO