UniTemp: Unlocking Video Generation in Any Temporal Order via Bidirectional Distillation

2026-06-17

AI Summary

Existing autoregressive video diffusion models are constrained by forward-only temporal generation, limiting their ability to support flexible creative tasks such as backward extension or mid-sequence frame interpolation. This work proposes UniTemp, the first unified autoregressive framework capable of generating videos in arbitrary temporal orders within a single model. By integrating a causal 3D VAE with a block-level anchor latent mechanism and a bidirectional distillation training strategy, UniTemp effectively mitigates inter-block discontinuities during reverse-time generation and overcomes the temporal-direction limitations inherent in conventional causal architectures. Experiments demonstrate that UniTemp maintains high-quality generation for both short and long videos while enabling diverse complex temporal editing capabilities—including bidirectional extension, intermediate frame insertion, looped video synthesis, scene transitions, and visual storytelling—significantly enhancing the flexibility and controllability of video generation.

Abstract

Autoregressive video diffusion models have emerged as a promising approach for long video generation, achieving strong performance in streaming settings. However, existing methods are restricted to forward temporal generation, whereas practical video creation often requires flexible generation order, e.g., conditioning on future context to extend backward, or on both past and future context for inbetween generation. We bridge this gap by training an autoregressive model that supports generation in arbitrary temporal directions. A key technical challenge arises from the Causal 3D VAE widely used in video diffusion models, which encodes latents strictly conditioned on past context. While suited for forward generation, this causal structure causes inter-block discontinuities when generation proceeds backward. To address this, we introduce blockwise anchor latents, a set of auxiliary latents that restore the missing past context at block boundaries during backward generation. Built on this design, we propose UniTemp, a bidirectional distillation framework that trains a single autoregressive student model for any-direction video generation. At inference time, UniTemp conditions on arbitrary past and/or future frames, improving controllability for both bidirectional and inbetween generation. Experiments show that UniTemp maintains competitive performance on short and long video generation compared to forward-only methods, while enabling diverse workflows such as bidirectional video extension, inbetween generation, looping video generation, scene transition, and visual story generation. Project website: https://lzhangbj.github.io/projects/unitemp/

Problem

• video generation
• temporal order
• autoregressive models
• bidirectional generation
• inbetweening

Innovation

• bidirectional video generation
• autoregressive diffusion model
• blockwise anchor latents
• temporal flexibility
• video inbetweening