Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker

2026-05-25

AI Summary

Existing referring expression comprehension (REC) benchmarks are confined to simple scenarios and assume a one-to-one correspondence between expressions and target objects, rendering them inadequate for the complexities of open-world settings. To address this limitation, this work introduces OpenRef—the first open-world REC benchmark that supports multi-target and no-target samples alongside diverse visual and linguistic modalities—and proposes a novel evaluation metric, N3R. Furthermore, the authors design a training-agnostic, plug-and-play Multi-task Consistency Checker (MCC) that enhances model robustness through a self-verification mechanism during inference. Experimental results demonstrate that MCC substantially improves the performance of existing REC models in complex scenarios, thereby validating both the challenge posed by OpenRef and the effectiveness of the proposed approach.

Abstract

Referring expression comprehension (REC) aims to localize a target object within an image based on a given expression. Although recent advances in vision-language models have led to substantial improvements in REC tasks, current REC benchmarks often hold simple scenarios and the assumption that each expression maps to a unique object. These limitations hinder the deployment of REC models in open-world environments. To fill this gap, we introduce OpenRef, a new benchmark for REC in complex visual and linguistic scenarios. OpenRef features three key advancements: 1) Diverse visual scenarios: spanning diverse visual domains, including ground views, drone views, dark scenes and adverse weather conditions; 2) Variable target counts: breaking the single-target limitation with multi-target and none-target samples; 3) Rich vocabulary types: incorporating proper nouns, polysemous words and ordinal terms to fit a wider range of expression needs. Furthermore, as traditional metrics are insufficient for open-world setting, we leverage F1 to measure grounding accuracy and propose N3R (Negative Relative Rejection Reliability) to assess relative rejection reliability against negative expressions. Finally, we introduce Multi-task Consistency Checker (MCC), a training-free but plug-and-play strategy that enhances model performance with one click by enforcing consistency self-verification. Extensive experiments demonstrate that this work significantly advances the performance of existing REC models in complex scenarios, paving the way for open-world REC. Project page: https://zongjianwu.github.io/openref

Problem

• Referring Expression Comprehension
• Open-world
• Visual Grounding
• Benchmark
• Multi-target

Innovation

• open-world referring expression comprehension
• Multi-task Consistency Checker
• training-free enhancement
• N3R metric
• variable target grounding