GREx: Generalized Referring Expression Segmentation, Comprehension, and Generation

2026-01-08

International Journal of Computer Vision

AI Summary

Existing referring expression tasks are confined to single-target scenarios and struggle to handle real-world cases involving multiple or no targets. This work proposes the Generalized Referring Expression task (GREx), which unifies segmentation, understanding, and generation for an arbitrary number of targets, and introduces gRefCOCO—the first large-scale dataset supporting GREx while remaining compatible with conventional benchmarks to assess performance gaps. To address this generalized setting, we design the ReLA baseline model, which incorporates a region-adaptive partitioning strategy and leverages sub-instance cues to explicitly model both region-to-region and region-to-language dependencies. Experiments demonstrate that ReLA achieves state-of-the-art performance on both GRES and GREC tasks, validating the effectiveness and challenge of the proposed benchmark.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Generation
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Referring Expression Segmentation (RES) and Comprehension (REC) respectively segment and detect the object described by an expression, while Referring Expression Generation (REG) generates an expression for the selected object. Existing datasets and methods commonly support single-target expressions only, i.e., one expression refers to one object, not considering multi-target and no-target expressions. This greatly limits the real applications of REx (RES/REC/REG). This paper introduces three new benchmarks called Generalized Referring Expression Segmentation (GRES), Comprehension (GREC), and Generation (GREG), collectively denoted as GREx, which extend the classic REx to allow expressions to identify an arbitrary number of objects. We construct the first large-scale GREx dataset gRefCOCO that contains multi-target, no-target, and single-target expressions and their corresponding images with labeled targets. GREx and gRefCOCO are designed to be backward-compatible with REx, facilitating extensive experiments to study the performance gap of the existing REx methods on GREx tasks. One of the challenges of GRES/GREC is complex relationship modeling, for which we propose a baseline ReLA that adaptively divides the image into regions with sub-instance clues and explicitly models the region-region and region-language dependencies.The proposed ReLA achieves the state-of-the-art results on both GRES and GREC tasks. The proposed gRefCOCO dataset and method are available at https://henghuiding.github.io/GRES.

Problem

• Referring Expression Segmentation
• Referring Expression Comprehension
• Referring Expression Generation
• multi-target expressions
• no-target expressions

Innovation

• Generalized Referring Expression
• Multi-target Referring
• Region-Language Dependency Modeling
• ReLA
• gRefCOCO