VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders

2026-07-15

AI Summary

Existing 3D variational autoencoders struggle to effectively model the semantic content and spatiotemporal structure of videos, and the rich representations from frozen video foundation models (VFMs) have not yet been efficiently transformed into compact, generation-friendly latent spaces. This work proposes VideoRAE, which demonstrates for the first time that multiscale features from a frozen VFM can be efficiently compressed—via a lightweight 1D self-attention projector—into either continuous or discrete latent representations. To enhance semantic fidelity during decoding, VideoRAE introduces a local-global alignment objective. Notably, the method operates without KL regularization and is compatible with both diffusion and autoregressive architectures. It achieves state-of-the-art class-conditional video generation performance on UCF-101 (gFVD: 40 for AR, 93 for DiT), converges approximately five times faster than existing autoencoders, and significantly accelerates training convergence in 2B-scale text-to-video tasks.

Abstract

Video generative models commonly rely on latent spaces learned by 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, which can limit the semantic and spatio-temporal structure captured by their latents. Meanwhile, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 show strong video understanding capabilities, yet whether their frozen representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored. We answer this question with VideoRAE, a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention projector. VideoRAE supports both continuous latents for Diffusion Transformers and discrete tokens for autoregressive models via multi-codebook high-dimensional quantization. During decoding, a local-and-global representation alignment objective with the frozen VFM teacher improves semantic preservation and enables training without KL regularization. Experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, it obtains state-of-the-art class-to-video gFVDs of 40 and 93 with AR and DiT generators, respectively, while converging approximately 5x faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE leads to faster convergence under comparable settings. These results validate frozen VFM representations as versatile and generation-friendly video latents. The model and code will be released on https://zhxie0117.github.io/VideoRAE.

Problem

• Video Foundation Models
• Representation Autoencoders
• Generative Modeling
• Latent Space
• Video Generation

Innovation

• Video Foundation Models
• Representation Autoencoder
• Latent Compression
• Multi-scale Feature Alignment
• Discrete-Continuous Latents