URECA: Unique Region Caption Anything

2025-04-07

AI Summary

Existing image region captioning methods suffer from redundancy and ambiguity when generating unique natural language descriptions across multiple granularities (e.g., parts, background). This paper proposes the first stage-wise, MLLM-driven framework explicitly designed for uniqueness-aware description generation. It introduces a dynamic masking modeling mechanism to explicitly disentangle regional semantics, incorporates a high-resolution mask encoder to enhance spatial awareness, and establishes an end-to-end data distillation pipeline for high-quality multi-granularity annotation. Evaluated on the newly constructed URECA dataset, the method achieves state-of-the-art performance; it also demonstrates significant cross-domain generalization gains on RefCOCO+ and RefCOCOg. Key contributions include: (1) the first MLLM training paradigm supporting fine-grained uniqueness-aware captioning; and (2) a novel mask modeling architecture that jointly optimizes spatial precision and semantic discriminability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Region-level captioning aims to generate natural language descriptions for specific image regions while highlighting their distinguishing features. However, existing methods struggle to produce unique captions across multi-granularity, limiting their real-world applicability. To address the need for detailed region-level understanding, we introduce URECA dataset, a large-scale dataset tailored for multi-granularity region captioning. Unlike prior datasets that focus primarily on salient objects, URECA dataset ensures a unique and consistent mapping between regions and captions by incorporating a diverse set of objects, parts, and background elements. Central to this is a stage-wise data curation pipeline, where each stage incrementally refines region selection and caption generation. By leveraging Multimodal Large Language Models (MLLMs) at each stage, our pipeline produces distinctive and contextually grounded captions with improved accuracy and semantic diversity. Building upon this dataset, we present URECA, a novel captioning model designed to effectively encode multi-granularity regions. URECA maintains essential spatial properties such as position and shape through simple yet impactful modifications to existing MLLMs, enabling fine-grained and semantically rich region descriptions. Our approach introduces dynamic mask modeling and a high-resolution mask encoder to enhance caption uniqueness. Experiments show that URECA achieves state-of-the-art performance on URECA dataset and generalizes well to existing region-level captioning benchmarks.

Problem

• Generates unique captions for multi-granularity image regions
• Addresses lack of diverse region-caption mappings in existing datasets
• Enhances caption accuracy and semantic diversity using MLLMs

Innovation

• Stage-wise data curation pipeline for captions
• Dynamic mask modeling for unique captions
• High-resolution mask encoder for fine details