VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

2026-04-14

AI Summary

This work addresses the limitations of conventional video tokenization methods, which rely on fixed-length 3D grids and force models to learn fine-grained details pixel by pixel, resulting in high training complexity and poor adaptability to videos of varying lengths or complexities. To overcome these challenges, the authors propose a flexible, coarse-to-fine hierarchical video tokenization approach that integrates a semantics-aware adaptive token allocation mechanism with a generative flow decoder. This framework enables end-to-end trainable video representations with on-demand token count adjustment—the first of its kind. The method substantially improves training efficiency and reduces model size: using only 1.1B parameters (with just one-eighth the token count of traditional approaches), it generates high-quality 10-second, 81-frame videos in both text-to-video and class-conditional generation tasks, matching the performance of models as large as 5.2B parameters.

Technology Category

• Computer Vision: Large Vision Models
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video's inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner -- where the first tokens (emergently) capture abstract information, such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count according to downstream needs and encoding videos longer than the baselines with the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.

Problem

• video tokenization
• generative modeling
• computational efficiency
• long video generation
• representation learning

Innovation

• flexible-length tokenization
• coarse-to-fine representation
• video tokenization
• generative flow decoder
• efficient video generation