QWERTY: Training-Free Motion Control via Query-Warped Video Diffusion Transformers

2026-07-02

AI Summary

This work addresses the limitation in video diffusion Transformers where motion control typically relies on text prompts or costly fine-tuning. The authors propose a training-free, explicit motion control method that leverages user-defined object warping and optical flow to impose frame-invariant semantic distortions on the query subspace of a pretrained image-to-video DiT, thereby steering the diffusion trajectory. This approach is further enhanced by a latent self-guidance optimization strategy based on predicted noise, enabling high-precision motion manipulation. Notably, it achieves state-of-the-art performance—comparable to fine-tuned methods—while preserving generation quality and stability, and represents the first demonstration of training-free motion control within a DiT architecture equipped with full 3D attention.

Abstract

Video diffusion transformers (DiTs) generate high-fidelity and temporally coherent videos, yet motion control remains implicit, primarily relying on text prompts. As a result, achieving desired motion often requires extensive prompt engineering and repeated resampling. While fine-tuning models with additional spatial prompts (e.g., bounding boxes or point trajectories) enables explicit control, it demands substantial data curation and computation, and may compromise the generative capabilities of pretrained models. Consequently, training-free motion control using such spatial prompts has been explored in U-Net-based video diffusion models, but remains largely unexplored for DiTs. We introduce QWERTY, a training-free framework that enables flexible motion control in pretrained image-to-video DiTs via user-defined object warping and optical flow. We carefully manipulate the 3D full attention of DiTs by warping the frame-invariant semantic subspace of queries. We find that the noise predicted by the query-warped DiT naturally guides the diffusion trajectory toward the desired motion, and further show that leveraging this noise as self-guidance for latent optimization improves control stability and visual quality. Experiments show that QWERTY achieves the most effective motion control among existing training-free approaches on a recent image-to-video DiT, with performance comparable to fine-tuning-based methods.

Problem

• motion control
• video diffusion transformers
• training-free
• spatial prompts
• object warping

Innovation

• training-free motion control
• video diffusion transformers
• query warping
• optical flow
• latent optimization