Can Segmentation Models Understand the World? Towards Proactive Affordance Reasoning via Visual Chain-of-Thought

2026-05-26

AI Summary

This work addresses the challenge that existing segmentation models struggle with high-level instructions that convey intent without explicitly specifying target regions. To this end, the authors propose SegWorld, a novel framework that introduces an active visual chain-of-thought mechanism: it proactively observes the scene prior to receiving any instruction to infer potential object behaviors, and upon receiving an intent-based command, performs multi-level reasoning—from target object to action to interacting part—to achieve functionality-aware segmentation. The method models scene context as linguistic priors and integrates large language models with a mask decoder, formalizing the visual chain of thought through probabilistic inference. The authors also establish the first affordance-based segmentation benchmark linking intent to object parts. Experiments demonstrate that SegWorld matches state-of-the-art performance on referential instructions while significantly outperforming existing approaches on intent-level commands.

Abstract

Recent segmentation models couple large language models (LLMs) with mask decoders to ground complex language expressions into masks, yet their instructions remain target-referential: they describe, constrain, or imply the region to be segmented. However, in real-world embodied interaction, human instructions are often at the intent-level, which includes the desired outcome without naming the region that enables it. To bridge this gap, we introduce SegWorld, where the model reasons about the scene through a multi-level visual chain-of-thought (CoT) before committing to a mask. Before receiving any instructions, it proactively observes the scene, describing visible objects and inferring plausible events they may support. Given an instruction, it continues the chain: from the object relevant to the intent, through the action that satisfies it, to the physical interaction site, the object part that affords the action. We formalize SegWorld as probabilistic inference, in which proactive observation supplies a linguistic scene context that improves mask prediction when instructions are given at the level of intent. We construct an intent-to-part benchmark for evaluating affordance-bearing part segmentation from high-level goals. Experiments show SegWorld matches instruction-driven baselines on target-referential instructions and improves substantially on intent-level ones.

Problem

• affordance reasoning
• intent-level instruction
• part segmentation
• visual chain-of-thought
• embodied interaction

Innovation

• visual chain-of-thought
• affordance reasoning
• intent-level segmentation
• proactive scene understanding
• part segmentation