Spatially-Grounded Text-to-Video Generation via Inference-Time Gradient-Free Optimization

2026-08-13

AI Summary

Existing text-to-video generation models rely on computationally expensive gradient-based optimization to achieve fine-grained spatial controllability. This work proposes a training-free, gradient-free inference-time optimization method that introduces, for the first time in this task, an analytical trajectory control mechanism. By explicitly solving cross-attention scores within diffusion Transformers and dynamically injecting spatial guidance signals based on the latent space manifold structure, the approach circumvents backpropagation entirely. It achieves substantially improved object localization accuracy over current baselines while incurring only minimal additional computational overhead.

Abstract

Diffusion Transformer Text-to-Video models have achieved remarkable synthesis quality, yet fine-grained spatial controllability remains a significant challenge. While existing training-free methods produce solid overall results in spatially grounded generation, \ie, placing a specific object in a designated location, they rely on gradient-based optimization techniques that incur prohibitive computational overhead, a bottleneck amplified in modern large-scale architectures. To address this limitation, we present Gradient-free Analytical Trajectory Optimization Video Generation (GATO-Vid), a novel training-free and gradient-free approach for precise spatial guidance. Rather than relying on costly backward passes, we introduce an alternative cross-attention score and solve it analytically to obtain an exact, closed-form solution. To use our analytical solution, we propose an on-the-fly injection mechanism tailored to the topological manifold of the transformer's latent space. Our experiments demonstrate that GATO-Vid significantly outperforms existing baselines in localization accuracy while introducing minimal computational overhead.

Problem

• spatially-grounded generation
• text-to-video
• spatial controllability
• computational overhead
• gradient-based optimization

Innovation

• gradient-free optimization
• spatially-grounded generation
• text-to-video
• diffusion transformer
• analytical solution