GroundingSuite: Measuring Complex Multi-Granular Pixel Grounding

2025-03-13

AI Summary

Existing referring expression segmentation datasets suffer from three major bottlenecks: narrow category coverage, limited linguistic diversity in expressions, and low annotation quality. To address these, this work introduces the first benchmark suite for multi-granularity pixel-level vision-language grounding. We propose a novel multi-VLM agent collaboration framework for automated annotation, achieving 4.5× higher annotation efficiency than GLaMM while significantly improving accuracy. We release a large-scale training set comprising 9.56 million expression-segmentation pairs and a high-quality evaluation benchmark of 3,800 images—covering fine-grained localization and cross-scene generalization. Our model achieves state-of-the-art performance with 68.9 cIoU on gRefCOCO and 55.3 gIoU on RefCOCOm, substantially outperforming prior methods. This work establishes a scalable, high-fidelity, and multi-granularity paradigm for pixel-level vision-language understanding.

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Pixel grounding, encompassing tasks such as Referring Expression Segmentation (RES), has garnered considerable attention due to its immense potential for bridging the gap between vision and language modalities. However, advancements in this domain are currently constrained by limitations inherent in existing datasets, including limited object categories, insufficient textual diversity, and a scarcity of high-quality annotations. To mitigate these limitations, we introduce GroundingSuite, which comprises: (1) an automated data annotation framework leveraging multiple Vision-Language Model (VLM) agents; (2) a large-scale training dataset encompassing 9.56 million diverse referring expressions and their corresponding segmentations; and (3) a meticulously curated evaluation benchmark consisting of 3,800 images. The GroundingSuite training dataset facilitates substantial performance improvements, enabling models trained on it to achieve state-of-the-art results. Specifically, a cIoU of 68.9 on gRefCOCO and a gIoU of 55.3 on RefCOCOm. Moreover, the GroundingSuite annotation framework demonstrates superior efficiency compared to the current leading data annotation method, i.e., $4.5 imes$ faster than the GLaMM.

Problem

• Limited object categories in existing datasets
• Insufficient textual diversity in current datasets
• Scarcity of high-quality annotations for pixel grounding

Innovation

• Automated data annotation using Vision-Language Models
• Large-scale dataset with diverse referring expressions
• Curated benchmark for precise evaluation