RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension

2025-12-05

AI Summary

Existing referring expression comprehension (REC) benchmarks emphasize perceptual evaluation and lack interpretable assessment of multi-level cognitive capabilities in multimodal large language models (MLLMs). Method: We propose PERC—the first REC benchmark explicitly designed to evaluate both perception and reasoning—by decoupling the task into six progressively challenging categories: attribute, location, interaction, commonsense, relation, and refusal. We introduce a dual-dimensional, six-task evaluation framework; design an end-to-end automated image–text pairing generation pipeline; and innovate a dynamic IoU-driven GRPO reinforcement learning strategy to enhance localization accuracy under complex reasoning. Contribution/Results: Experiments demonstrate that PERC significantly increases benchmark difficulty and interpretability, enabling fine-grained, cognitively grounded evaluation of MLLMs. It establishes a novel paradigm for assessing hierarchical reasoning and grounded understanding in vision-language models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Referring Expression Comprehension (REC) is a vision-language task that localizes a specific image region based on a textual description. Existing REC benchmarks primarily evaluate perceptual capabilities and lack interpretable scoring mechanisms, which cannot reveal the grounding capability of Multi-modal Large Language Model (MLLM) across different cognitive abilities. To address this limitation, we introduce RefBench-PRO, a comprehensive REC benchmark, which decomposes referring expressions into two core dimensions, i.e., perception and reasoning, and further subdivides them into six progressively challenging tasks, such as attribute, position, interaction, commonsense, relation and reject. We also develop a fully automated data-generation pipeline that produces diverse referring expressions across these six sub-dimensions. Furthermore, We propose Ref-R1, an RL-based learning scheme, which incorporates Dynamic IoU-based GRPO to improve localization accuracy under increasingly complex reasoning conditions, establishing a stronger baseline for REC. Extensive experiments demonstrate that our RefBench-PRO enables interpretable evaluation of MLLM on referring expression comprehension, presenting greater challenges in both perception and reasoning.

Problem

• Develops a benchmark to evaluate multimodal models' perceptual and reasoning abilities in referring expression comprehension
• Decomposes referring expressions into six progressively challenging tasks for interpretable scoring
• Proposes an RL-based method to improve localization accuracy under complex reasoning conditions

Innovation

• Decomposes referring expressions into perception and reasoning dimensions
• Develops automated data-generation pipeline for diverse sub-tasks
• Proposes RL-based learning scheme with Dynamic IoU-GRPO for localization