Task-Oriented Data Synthesis and Control-Rectify Sampling for Remote Sensing Semantic Segmentation

2025-12-18

AI Summary

In remote sensing semantic segmentation, synthetic labeled data suffer from poor controllability, low mask accuracy, and unstable sampling quality—limiting downstream task performance. To address this, we propose TODSynth, a task-oriented data synthesis framework. Methodologically, we design a text–image–mask joint multimodal attention mechanism and build a Multimodal Diffusion Transformer (MM-DiT) based on the DiT architecture. We further introduce Control-aware Rectified Flow Matching (CRFM), which dynamically calibrates the sampling trajectory early in generation using semantic loss, and employ full-branch fine-tuning guided by task-specific feedback. Experiments demonstrate that TODSynth significantly improves task adaptability and generation stability—especially under few-shot and complex-scene conditions—achieving superior mask accuracy and segmentation performance compared to existing controllable generation methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

With the rapid progress of controllable generation, training data synthesis has become a promising way to expand labeled datasets and alleviate manual annotation in remote sensing (RS). However, the complexity of semantic mask control and the uncertainty of sampling quality often limit the utility of synthetic data in downstream semantic segmentation tasks. To address these challenges, we propose a task-oriented data synthesis framework (TODSynth), including a Multimodal Diffusion Transformer (MM-DiT) with unified triple attention and a plug-and-play sampling strategy guided by task feedback. Built upon the powerful DiT-based generative foundation model, we systematically evaluate different control schemes, showing that a text-image-mask joint attention scheme combined with full fine-tuning of the image and mask branches significantly enhances the effectiveness of RS semantic segmentation data synthesis, particularly in few-shot and complex-scene scenarios. Furthermore, we propose a control-rectify flow matching (CRFM) method, which dynamically adjusts sampling directions guided by semantic loss during the early high-plasticity stage, mitigating the instability of generated images and bridging the gap between synthetic data and downstream segmentation tasks. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art controllable generation methods, producing more stable and task-oriented synthetic data for RS semantic segmentation.

Problem

• Enhance remote sensing semantic segmentation via synthetic data generation
• Improve control and quality in synthetic data for segmentation tasks
• Address few-shot and complex-scene challenges in remote sensing data

Innovation

• Multimodal Diffusion Transformer with unified triple attention
• Control-rectify flow matching guided by semantic loss
• Task-oriented data synthesis framework for segmentation