FlowSeg: Dynamic Semantic Guidance for LLM-Conditioned Segmentation

2026-05-28

AI Summary

Existing large language model (LLM)-guided segmentation approaches treat linguistic semantics as static prompts or post-hoc signals, which hinders precise alignment between language and segmentation masks. To address this limitation, this work proposes FlowSeg, introducing a novel dynamic semantic guidance mechanism. Within an end-to-end iterative segmentation framework, FlowSeg enables real-time bidirectional semantic interaction that jointly optimizes language conditions and visual decoding states. Additionally, a lightweight boundary-aware refinement strategy is incorporated to enhance accuracy in ambiguous regions. This approach substantially improves semantic alignment between language expressions and predicted masks, achieving state-of-the-art performance on both referring expression segmentation and reasoning-based segmentation benchmarks.

Abstract

LLM-conditioned segmentation has recently advanced rapidly by coupling large language models with iterative mask generation frameworks. However, we identify a persistent failure mode in current propose-then-select pipelines. Although high-quality mask candidates are often generated, the final prediction may fail to match the given linguistic condition. This failure arises because language semantics are typically used as static prompts or post-hoc matching signals, rather than participating in the iterative mask generation process. Through systematic analysis, we show that many errors stem from semantic misalignment rather than poor mask quality. To address this issue, we propose FlowSeg, which introduces dynamic semantic guidance via a bidirectional semantic flow between intermediate decoding states and LLM-derived condition embeddings throughout the generation process. Language conditions actively guide mask refinement at each stage, while condition embeddings are progressively updated by emerging visual evidence. This design yields semantically grounded mask representations and visually aligned language conditions, enabling more reliable matching. We further incorporate a lightweight boundary-aware refinement to selectively enhance uncertain regions without perturbing confident interiors. Extensive experiments on referring expression segmentation and reasoning segmentation tasks demonstrate that FlowSeg consistently improves language-mask alignment and achieves state-of-the-art performance. Project page: https://zkzhang98.github.io/FlowSeg_page

Problem

• LLM-conditioned segmentation
• semantic alignment
• iterative mask generation
• language-mask mismatch
• referring expression segmentation

Innovation

• dynamic semantic guidance
• bidirectional semantic flow
• LLM-conditioned segmentation
• iterative mask refinement
• language-mask alignment