DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning

2025-06-11

arXiv.org

AI Summary

Natural language query-based GUI element localization faces challenges including visual element diversity, spatial clutter, and linguistic ambiguity. This paper proposes a training-free modality-separation and dynamic focusing framework: it decouples textual and icon representations, leverages a general-purpose vision-language model, and integrates candidate region generation with a progressive zooming mechanism to enable independent, modality-specific reasoning. Furthermore, hierarchical visual focusing is introduced, wherein spatial attention dynamically refines the localization region. To our knowledge, this is the first approach achieving modality-aware semantic alignment and fine-grained localization for GUI elements without any task-specific training. Evaluated on standard GUI benchmarks, our method significantly outperforms existing baselines, demonstrating both the effectiveness and generalizability of the co-designed modality separation and region focusing strategy.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Grounding natural language queries in graphical user interfaces (GUIs) poses unique challenges due to the diversity of visual elements, spatial clutter, and the ambiguity of language. In this paper, we introduce DiMo-GUI, a training-free framework for GUI grounding that leverages two core strategies: dynamic visual grounding and modality-aware optimization. Instead of treating the GUI as a monolithic image, our method splits the input into textual elements and iconic elements, allowing the model to reason over each modality independently using general-purpose vision-language models. When predictions are ambiguous or incorrect, DiMo-GUI dynamically focuses attention by generating candidate focal regions centered on the model's initial predictions and incrementally zooms into subregions to refine the grounding result. This hierarchical refinement process helps disambiguate visually crowded layouts without the need for additional training or annotations. We evaluate our approach on standard GUI grounding benchmarks and demonstrate consistent improvements over baseline inference pipelines, highlighting the effectiveness of combining modality separation with region-focused reasoning.

Problem

• Grounding natural language queries in diverse GUI elements
• Disambiguating visually crowded and spatially cluttered layouts
• Resolving language ambiguity without training or annotations

Innovation

• Modality-aware visual reasoning for GUI grounding
• Dynamic region focusing without additional training
• Hierarchical refinement for disambiguating crowded layouts