ProxyUp: Training-Free Proxy-Conditioned Video Generation for Controllable Dynamics

2026-07-04

AI Summary

Existing video generation models struggle to precisely control complex, physically plausible dynamics and object interactions through text alone. This work proposes a training-free proxy-conditioned video generation approach that leverages coarse-grained proxy videos—derived from physics simulations or real-world recordings—as dynamic priors to synthesize novel content with temporally coherent motion guided by text prompts. Built upon a pretrained video diffusion model, the method integrates proxy dynamics and textual semantics through latent-space inversion of the proxy video, region-aware latent noise injection, and a Stochastic Flow Relaxation (SFR) mechanism. Experiments demonstrate that the proposed approach significantly outperforms current video editing and motion transfer techniques in both dynamic fidelity and alignment with textual descriptions.

Abstract

Precise control over complex dynamics remains challenging for modern video generative models, as text prompts alone often cannot specify physically plausible, fine-grained motion and interactions. We introduce $\textit{proxy-conditioned video generation}$, where a coarse proxy video from physics-based simulation or real-world recording serves as a dynamics carrier to control foreground object motion. Given a proxy video and a text prompt, the goal is to synthesize a new video that preserves the proxy dynamics while generating novel content and plausible interactions aligned with the prompt. Since paired proxy-target videos are difficult to obtain, we propose $\textbf{ProxyUp}$, a training-free framework built on pretrained video generative models. ProxyUp first inverts the proxy video into an intermediate latent representation and applies $\textbf{region-wise latent noising}$, preserving motion-critical proxy latents while injecting noise into regions intended for text-driven regeneration. To mitigate the distribution mismatch and weak foreground-background coupling introduced by this heuristic latent composition, we further propose $\textbf{Stochastic Flow Relaxation (SFR)}$, which progressively relaxes the composed latent toward the model's learned distribution before ODE sampling. Experiments on both simulation and real-world proxies show that ProxyUp outperforms strong video editing and motion transfer baselines in dynamic fidelity and text alignment.

Problem

• video generation
• controllable dynamics
• motion control
• text-to-video
• dynamic fidelity

Innovation

• proxy-conditioned video generation
• training-free
• region-wise latent noising
• Stochastic Flow Relaxation
• controllable dynamics