Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elements

2026-03-15

AI Summary

This work proposes ZoomUI, a training-free approach for GUI grounding that circumvents the high cost and data-quality limitations of conventional methods relying on extensive labeled datasets to fine-tune multimodal large language models. ZoomUI leverages inference-time scaling to progressively anchor natural language instructions to interface elements through a synergistic combination of latent reasoning optimization, internal attention mechanisms, and iterative region zooming. This enables a stepwise visual focus refinement process without any model training. Evaluated across multiple benchmarks, ZoomUI achieves performance on par with or superior to state-of-the-art methods while drastically reducing dependence on annotated data, thereby establishing the first training-free GUI interaction paradigm grounded in inference-time scaling.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Multimodal Large Language Model (MLLM)-based Graphical User Interface (GUI) agents develop rapidly, with visual grounding that maps natural language instructions to target UI elements serving as the core capability. Existing GUI agents typically fine-tune MLLM on massive datasets to handle challenges in understanding instructions and UI interfaces, which not only incurs high data annotation costs but also makes performance dependent on data quality and distribution. To avoid such cumbersome yet ineffective training, we notice that complex UI interfaces can be decomposed into basic visual elements directly understandable by common MLLMs. Consequently, we propose ZoomUI that leverages inference scaling to guide common MLLMs in progressively anchor instruction elements to increasingly detailed interface elements. Specifically, ZoomUI first optimizes the latent thinking to transform original instruction into element visual features description, and subsequently leverages internal attention to iteratively zoom in target element interface region. Evaluations on extensive benchmarks demonstrate that ZoomUI reaches or even surpasses SOTA baselines.

Problem

• GUI grounding
• Multimodal Large Language Model
• visual grounding
• data annotation cost
• instruction-to-UI mapping

Innovation

• trainless GUI grounding
• inference scaling
• multimodal LLM
• visual grounding
• iterative attention