Enhancing Video Physical Consistency via Role-aware Joint Training and Modality-decoupled Denoising

2026-07-06

AI Summary

Existing video diffusion models struggle to maintain long-term physical consistency. To address this, this work proposes a character-aware grouping mechanism to distinguish entity types and introduces a modality-decoupled denoising strategy that trains visual and auxiliary modalities—such as optical flow—at independent noise levels. Furthermore, it incorporates loss-weight decay and a cross-step self-guidance mechanism, leveraging auxiliary modalities as soft constraints to effectively mitigate error accumulation during inference. Evaluated on the VideoPhy benchmark, the proposed approach achieves a 39.4% improvement in the SA metric and a 17.9% gain in the PC metric over the Wan2.1-T2V-1.3B model, with consistent enhancements observed on VideoPhy-2, demonstrating a balanced advancement in both generation quality and physical plausibility.

Abstract

While modern video diffusion models excel in visual fidelity, maintaining long-range physical consistency remains a formidable challenge. Conventional pixel-reconstruction objectives mainly focus on appearance details and often fail to capture the underlying dynamics of a scene. To mitigate this, recent efforts have integrated auxiliary modalities (e.g., optical flow) to introduce physics priors via joint training with video appearance. However, these methods have three main limitations: (1) they do not distinguish the different motion patterns of different entity types; (2) joint modeling of visual and auxiliary modalities can cause capacity conflicts and weaken the pretrained visual prior; and (3) auxiliary modalities may accumulate errors during inference. To address these issues, we propose \textbf{VPT}, a fine-tuning framework for improving physical consistency in video diffusion models. VPT introduces a role-aware signal that groups entities into agents, controlled objects, passive objects, and background, so that different physical roles can be modeled more clearly. We further propose a modality-decoupled denoising strategy, where the visual and auxiliary channels are assigned independent noise levels. Together with a loss-weight decay strategy, this design makes auxiliary modalities serve as soft constraints rather than strong dependencies, mitigating recursive prediction errors during inference. We also introduce cross-step auto-guidance to further strengthen physical dynamics. Experiments show that VPT improves physical consistency while preserving visual quality, achieving relative gains of 39.4\% in SA and 17.9\% in PC on VideoPhy benchmark over Wan2.1-T2V-1.3B, and consistent improvements on VideoPhy-2 benchmark. The project page is available at https://tom-zgt.github.io/VPT.

Problem

• physical consistency
• video diffusion models
• auxiliary modalities
• motion patterns
• inference error accumulation

Innovation

• role-aware modeling
• modality-decoupled denoising
• video diffusion models
• physical consistency
• cross-step auto-guidance