Conversational Image Segmentation: Grounding Abstract Concepts with Scalable Supervision

2026-02-13

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Conversational AI/Dialog Systems
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Conversational image segmentation grounds abstract, intent-driven concepts into pixel-accurate masks. Prior work on referring image grounding focuses on categorical and spatial queries (e.g.,"left-most apple") and overlooks functional and physical reasoning (e.g.,"where can I safely store the knife?"). We address this gap and introduce Conversational Image Segmentation (CIS) and ConverSeg, a benchmark spanning entities, spatial relations, intent, affordances, functions, safety, and physical reasoning. We also present ConverSeg-Net, which fuses strong segmentation priors with language understanding, and an AI-powered data engine that generates prompt-mask pairs without human supervision. We show that current language-guided segmentation models are inadequate for CIS, while ConverSeg-Net trained on our data engine achieves significant gains on ConverSeg and maintains strong performance on existing language-guided segmentation benchmarks. Project webpage: https://glab-caltech.github.io/converseg/

Problem

• conversational image segmentation
• abstract concepts
• functional reasoning
• physical reasoning
• intent grounding

Innovation

• Conversational Image Segmentation
• Language-Guided Segmentation
• Affordance Reasoning
• AI-Powered Data Engine
• ConverSeg-Net