What Makes Synthetic Data Effective in Image Segmentation

2026-05-18

AI Summary

This work addresses the underutilization of synthetic data in complex image segmentation tasks by systematically investigating the impact of diffusion model–generated synthetic data on segmentation performance. The study identifies scene density and instance fidelity as critical factors for enhancing segmentation representations and introduces SENSE, a model-agnostic and scalable unified framework that requires no modifications to existing architectures. SENSE effectively boosts the performance of prominent segmentation models such as DPT and Mask2Former. Extensive experiments on Cityscapes, COCO, and ADE20K demonstrate that SENSE significantly improves segmentation accuracy, confirming its effectiveness and strong cross-dataset generalization capability.

Abstract

Driven by rapid advances in large-scale generative models, synthetic data has emerged as a promising solution for visual understanding. While modern diffusion models achieve remarkable photorealistic image synthesis, their potential in complex visual segmentation tasks remains underexplored. In this work, we conduct a systematic analysis of synthetic images from state-of-the-art diffusion models to uncover the factors governing their utility. In particular, synthetic images characterized by dense scene composition and fine instance fidelity demonstrate distinctive benefits, yielding significantly more discriminative spatial representations. Building on these insights, we propose SENSE, a unified framework that leverages flexible and scalable synthetic data to substantially enhance segmentation performance. Notably, SENSE is model-agnostic, compatible with diverse architectures (e.g., DPT and Mask2Former), and scales effectively across models with varying parameter capacities. Extensive experiments on Cityscapes, COCO, and ADE20K validate the effectiveness and generalization capability of our approach. Code is available at https://github.com/zhang0jhon/SENSE.

Problem

• synthetic data
• image segmentation
• diffusion models
• visual understanding
• spatial representations

Innovation

• synthetic data
• image segmentation
• diffusion models
• dense scene composition
• model-agnostic framework