Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs

2025-10-21

AI Summary

Current region-level multimodal large language models (MLLMs) process local regions in isolation, neglecting global visual context—thus hindering fine-grained analysis and multi-region relational reasoning in complex scenes. To address this, we propose GAR (Global-Aware Region understanding), a novel framework featuring: (i) an RoI-aligned feature replay mechanism that explicitly injects global visual context into region-level understanding; (ii) a context-enhanced visual encoder and a multi-region prompt interaction module supporting open-ended question answering and compositional reasoning. We further introduce GAR-Bench, the first benchmark systematically evaluating complex region-relational and compositional reasoning capabilities. Experiments show that GAR-1B outperforms DAM-3B by +4.5 on DLC-Bench; GAR-8B achieves zero-shot transfer superior to fine-tuned VideoRefer-7B on VideoRefer-BenchQ. These results mark a significant advance from holistic image description toward active, interactive visual understanding in MLLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

While Multimodal Large Language Models (MLLMs) excel at holistic understanding, they struggle in capturing the dense world with complex scenes, requiring fine-grained analysis of intricate details and object inter-relationships. Region-level MLLMs have been a promising step. However, previous attempts are generally optimized to understand given regions in isolation, neglecting crucial global contexts. To address this, we introduce Grasp Any Region (GAR) for comprehen- sive region-level visual understanding. Empowered by an effective RoI-aligned feature replay technique, GAR supports (1) precise perception by leveraging necessary global contexts, and (2) modeling interactions between multiple prompts. Together, it then naturally achieves (3) advanced compositional reasoning to answer specific free-form questions about any region, shifting the paradigm from passive description to active dialogue. Moreover, we construct GAR-Bench, which not only provides a more accurate evaluation of single-region comprehension, but also, more importantly, measures interactions and complex reasoning across multiple regions. Extensive experiments have demonstrated that GAR-1B not only maintains the state-of-the-art captioning capabilities, e.g., outperforming DAM-3B +4.5 on DLC-Bench, but also excels at modeling relationships between multiple prompts with advanced comprehension capabilities, even surpassing InternVL3-78B on GAR-Bench-VQA. More importantly, our zero-shot GAR-8B even outperforms in-domain VideoRefer-7B on VideoRefer-BenchQ, indicating its strong capabilities can be easily transferred to videos.

Problem

• Addressing fine-grained region understanding in complex visual scenes
• Overcoming limitations of isolated region analysis without global context
• Enabling active dialogue and compositional reasoning about specific regions

Innovation

• RoI-aligned feature replay technique for global contexts
• Modeling interactions between multiple prompts
• Advanced compositional reasoning for region-level understanding