Seg-VAR: Image Segmentation with Visual Autoregressive Modeling

2025-11-16

AI Summary

Traditional discriminative models for image segmentation suffer from insufficient modeling of low-level spatial structures. To address this, we propose Seg-VAR—the first generative segmentation framework that introduces Vision Autoregressive (VAR) modeling into dense prediction tasks. Our method reformulates segmentation as a conditional mask generation problem: an image encoder and a spatially aware seglat encoder jointly align the image–mask latent distributions, while a position-sensitive color mapping discretizes masks into sequence tokens to enable hierarchical, fine-grained prediction. A multi-stage training strategy enhances stability in implicit representation learning. Evaluated on PASCAL VOC and COCO-Stuff benchmarks, Seg-VAR significantly outperforms state-of-the-art discriminative and generative approaches. These results demonstrate the effectiveness and scalability of autoregressive modeling for pixel-level generation tasks.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

While visual autoregressive modeling (VAR) strategies have shed light on image generation with the autoregressive models, their potential for segmentation, a task that requires precise low-level spatial perception, remains unexplored. Inspired by the multi-scale modeling of classic Mask2Former-based models, we propose Seg-VAR, a novel framework that rethinks segmentation as a conditional autoregressive mask generation problem. This is achieved by replacing the discriminative learning with the latent learning process. Specifically, our method incorporates three core components: (1) an image encoder generating latent priors from input images, (2) a spatial-aware seglat (a latent expression of segmentation mask) encoder that maps segmentation masks into discrete latent tokens using a location-sensitive color mapping to distinguish instances, and (3) a decoder reconstructing masks from these latents. A multi-stage training strategy is introduced: first learning seglat representations via image-seglat joint training, then refining latent transformations, and finally aligning image-encoder-derived latents with seglat distributions. Experiments show Seg-VAR outperforms previous discriminative and generative methods on various segmentation tasks and validation benchmarks. By framing segmentation as a sequential hierarchical prediction task, Seg-VAR opens new avenues for integrating autoregressive reasoning into spatial-aware vision systems. Code will be available at https://github.com/rkzheng99/Seg-VAR.

Problem

• Reformulating image segmentation as conditional autoregressive mask generation
• Replacing discriminative learning with latent learning for segmentation
• Integrating autoregressive reasoning into spatial-aware vision systems

Innovation

• Segmentation as conditional autoregressive mask generation
• Replacing discriminative learning with latent learning process
• Multi-stage training strategy for latent representation alignment