MMDiff: Extending Diffusion Transformers for Multi-Modal Generation

2026-06-15

AI Summary

This work addresses the limitation of conventional diffusion models, which discard rich perceptual representations accumulated during the denoising process and thus struggle to support multimodal dense prediction tasks. The authors propose extending a frozen diffusion Transformer into a multimodal generative system that jointly synthesizes images along with corresponding dense perceptual modalities—such as semantic segmentation, salient object detection, and depth maps—via lightweight decoder heads. Their approach innovatively leverages perceptual information embedded in the temporal distribution of the denoising trajectory, introducing a multi-timestep feature fusion scheme coupled with spatially adaptive aggregation and concept-driven attention for interpretable spatial guidance. Experiments demonstrate substantial improvements, including a 28.7% gain in mIoU for semantic segmentation, strong performance across multiple dense prediction tasks, and efficient generation of large-scale synthetic data.

Abstract

Diffusion transformers have demonstrated remarkable generative capabilities, yet the rich perceptual representations computed across their denoising trajectory are discarded once the content is rendered. We present MMDiff, a framework that transforms a frozen diffusion transformer into a multi-modal generative system that jointly produces images alongside any combination of dense perceptual modalities using lightweight decoder heads. Our central finding is that perceptual information is temporally distributed along the denoising trajectory, and that multi-timestep feature fusion with spatially varying aggregation weights is essential, improving semantic segmentation results by up to 28.7% mIoU over single-timestep extraction. We further adopt concept-driven attention extraction for interpretable spatial guidance, and show that frozen diffusion features are competitive with and complementary to state-of-the-art encoders such as DINOv3. By training only lightweight decoder heads on a frozen backbone, we achieve strong performance in semantic segmentation, salient object detection, and depth estimation, and demonstrate that this framework enables effective synthetic data generation at scale.

Problem

• multi-modal generation
• diffusion transformers
• perceptual representations
• dense modalities
• feature reuse

Innovation

• diffusion transformers
• multi-modal generation
• feature fusion
• frozen backbone
• perceptual modalities