DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding

2026-05-14

AI Summary

Accurately localizing user-referred interface elements in high-resolution, element-dense GUI screenshots remains challenging. This work proposes a training-free dynamic region search framework that emulates human visual perception mechanisms—focusing, shifting, and saccading—by integrating a lightweight UI-aware module with a Monte Carlo Tree Search (MCTS)-based action planner. The framework adaptively explores and prunes candidate regions, guided by a region-quality reward mechanism that seamlessly integrates into existing multimodal large language models. Evaluated on the ScreenSpot-Pro dataset, the approach improves GUI grounding accuracy by 14% for models such as Qwen2.5-VL-7B and UGround-V1-7B, significantly enhancing their generalization capabilities.

Abstract

GUI agents powered by Multimodal Large Language Models (MLLMs) have demonstrated impressive capability in understanding and executing user instructions. However, accurately grounding instruction-relevant elements from high-resolution screenshots cluttered with irrelevant UI components remains challenging for existing approaches. Inspired by how humans dynamically adjust their perceptual scope to locate task-related regions on complex screens, we propose DRS-GUI, a training-free dynamic region search framework for GUI grounding that can be seamlessly integrated into existing MLLMs. DRS-GUI introduces a lightweight UI Perceptor that performs three human-like perceptual actions (Focus, Shift, and Scatter) to progressively explore the interface and generate region proposals. To dynamically schedule these actions, we further design an Action Planner based on Monte Carlo Tree Search (MCTS). A region quality reward is employed to evaluate and select the highly instruction-relevant region, efficiently pruning redundant UI elements. Experiments demonstrate that DRS-GUI yields a 14\% improvement on ScreenSpot-Pro for general and GUI-specific MLLMs (Qwen2.5-VL-7B and UGround-V1-7B), significantly enhancing grounding performance and generalization.

Problem

• GUI grounding
• multimodal large language models
• high-resolution screenshots
• irrelevant UI components
• instruction-relevant elements

Innovation

• Dynamic Region Search
• Training-Free GUI Grounding
• Multimodal Large Language Models
• Monte Carlo Tree Search
• UI Perceptor