Modular Diffusion Models for Structured Visual Recognition

2026-06-21

AI Summary

This work proposes Modular Diffusion Models (MDM), a novel approach that introduces modular design into diffusion-based generative frameworks to address the limitations of traditional structured visual recognition methods in modeling prediction uncertainty and multimodal outputs in complex scenes. MDM decomposes structured information—such as category, location, and relational attributes—into independent modules, enabling joint distribution learning over heterogeneous output spaces comprising both continuous and discrete variables. Each module can be trained independently and flexibly composed, while collaboratively generating complete structured predictions during inference. Evaluated on object detection, instance segmentation, and scene graph generation tasks, MDM demonstrates significantly enhanced capability in capturing uncertainty and multimodality compared to conventional approaches.

Abstract

Traditional supervised methods for structured visual recognition tasks -- such as object detection, segmentation, and scene graph generation -- often produce deterministic, fixed outputs, limiting their ability to capture the inherent uncertainty in complex visual scenes. As a consequence, such point estimates are unable to capture the prediction uncertainty (or multi modality) intrinsic to these problems, often arising from natural ambiguities (e.g., ambiguity in size of partially occluded objects, local ambiguity of exact segmentation boundary, etc.) as well as noise and sparsity of training data. To address this limitation, we present Modular Diffusion Models (MDMs), a simple and novel framework that learns a distribution over structured outputs for a given input image. MDMs decompose the diffusion process into distinct, task-specific modules, each focused on capturing a different aspect of the structured information space, such as object categories, spatial locations, and inter-object relationships. This modular design allows each component to be learned independently, with seamless integration at inference without additional training. Furthermore, the modularity of MDMs enables the diffusion process to easily operate over the heterogeneous output space common in many structured learning tasks (e.g., a continuous bounding boxes and discrete class labels). Experimental results over three distinct structured tasks -- object detection, instance segmentation, and scene graph generation -- highlight the benefits of our proposed framework.

Problem

• structured visual recognition
• prediction uncertainty
• multi-modality
• ambiguous visual scenes
• heterogeneous output space

Innovation

• Modular Diffusion Models
• structured visual recognition
• uncertainty modeling
• heterogeneous output space
• multi-modality