Omni-Referring Image Segmentation

2025-12-07

AI Summary

This paper introduces the OmniRIS task, addressing highly generalizable multimodal image segmentation by jointly leveraging textual instructions and reference images annotated with masks, bounding boxes, or scribbles as unified prompts—thereby synergizing fine-grained linguistic semantics with precise visual spatial localization. To this end, we propose the first multimodal unified prompting paradigm and design OmniSegNet, an end-to-end trainable architecture incorporating cross-modal alignment and multi-prompt fusion mechanisms. Furthermore, we release OmniRef, a large-scale, multi-scenario benchmark dataset comprising 180K samples. Extensive experiments demonstrate that our method significantly outperforms existing approaches across diverse segmentation settings—including one-to-one and many-to-many configurations—and exhibits strong generalization capability, particularly in localizing rare objects and performing attribute-driven segmentation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

In this paper, we propose a novel task termed Omni-Referring Image Segmentation (OmniRIS) towards highly generalized image segmentation. Compared with existing unimodally conditioned segmentation tasks, such as RIS and visual RIS, OmniRIS supports the input of text instructions and reference images with masks, boxes or scribbles as omni-prompts. This property makes it can well exploit the intrinsic merits of both text and visual modalities, i.e., granular attribute referring and uncommon object grounding, respectively. Besides, OmniRIS can also handle various segmentation settings, such as one v.s. many and many v.s. many, further facilitating its practical use. To promote the research of OmniRIS, we also rigorously design and construct a large dataset termed OmniRef, which consists of 186,939 omni-prompts for 30,956 images, and establish a comprehensive evaluation system. Moreover, a strong and general baseline termed OmniSegNet is also proposed to tackle the key challenges of OmniRIS, such as omni-prompt encoding. The extensive experiments not only validate the capability of OmniSegNet in following omni-modal instructions, but also show the superiority of OmniRIS for highly generalized image segmentation.

Problem

• Proposes OmniRIS for highly generalized image segmentation
• Handles text and visual prompts like masks, boxes, scribbles
• Introduces OmniRef dataset and OmniSegNet baseline for evaluation

Innovation

• Introduces OmniRIS supporting text and visual omni-prompts
• Proposes OmniSegNet baseline for omni-prompt encoding challenges
• Constructs OmniRef dataset with comprehensive evaluation system