From Dense Prediction to Visual Editing: Structured Supervision for Unified Image and Video Creation

2026-08-13

AI Summary

This study addresses the limited editing precision in unified generative models caused by insufficient local structural and temporal supervision in semantic conditioning. We propose embedding depth and normal prediction as structured supervision within a shared MMDiT backbone, marking the first integration of such geometric priors for this purpose. Through decoupled injection, cross-context attention, and progressive training strategies, our approach effectively transfers perceptual knowledge to editing tasks. Experiments demonstrate that the model achieves a top overall score of 4.15 on unified system benchmarks. Furthermore, introducing dense supervision improves the OpenVE overall metric to 4.06 and elevates local editing performance from 3.92 to 4.18, significantly enhancing both structural preservation and temporal consistency.

Abstract

Unified image and video creation requires a model to follow diverse instructions while preserving identity, geometry, and temporal structure from visual context. However, semantic-only conditioning and creation-only training do not explicitly supervise the local structure needed for precise, temporally consistent editing. We therefore formulate depth and surface-normal prediction as image-form denoising targets, using these dense tasks as structured visual supervision within the same creation interface. Our framework decouples semantic interpretation from spatially aligned visual injection while sharing one multimodal diffusion transformer (MMDiT) backbone across all tasks. Mutual Context Attention (MCA), a paired-video data-construction procedure, and a progressive training curriculum then connect the learned structural cues to temporally localized editing and reference-conditioned creation. A single checkpoint obtains the highest overall score in the reported comparison of unified systems (4.15); adding dense supervision improves OpenVE Overall from 3.98 to 4.06 and Local Add from 3.92 to 4.18. These results support a deliberately bounded conclusion: perception-oriented dense supervision transfers useful structural knowledge to downstream creation, especially editing locality and preservation; we do not claim superiority as a standalone dense predictor.

Problem

• Unified Image and Video Creation
• Visual Editing
• Structured Supervision
• Temporal Consistency
• Local Structure Preservation

Innovation

• Structured Supervision
• Multimodal Diffusion Transformer
• Mutual Context Attention
• Dense Prediction
• Unified Creation