Bridging Video Understanding and Generation in a Unified Framework

2026-06-30

AI Summary

Video understanding and generation are challenging to unify due to their divergent objectives: the former requires compact semantic representations, while the latter demands fine-grained detail preservation and temporal consistency. To address this, this work proposes Vega, the first framework to jointly model both tasks within a unified architecture for video. Vega aligns textual and visual representations through a shared semantic vocabulary, employs an autoregressive model to predict semantic tokens of keyframes, and leverages these tokens to guide a diffusion model in generating high-resolution, temporally coherent video frames. This hybrid approach achieves state-of-the-art performance on both VBench (for generation) and VideoMME (for understanding), demonstrating the effectiveness and versatility of a unified framework for video understanding and generation.

Abstract

Recently, unified image generation and understanding have been extensively explored. However, extending such unified modeling paradigms to the video domain remains largely underexplored. A central challenge is that video understanding favors compact, discriminative semantic representations, whereas video generation requires dense signals that preserve visual details and temporal coherence. Videos naturally capture both spatial semantics and temporal dynamics, making them a more suitable modality for unified multimodal modeling compared to static images. In this paper, we propose Vega, a unified framework that bridges video understanding and generation. Vega leverages a shared vocabulary to jointly model text and visual representations and employs a hybrid architecture combining autoregressive (AR) prediction with diffusion-based rendering. Specifically, the AR model focuses on predicting semantically meaningful visual tokens for keyframes, providing a structured representation that guides the diffusion module in rendering dense, high-resolution video frames. Extensive experiments demonstrate that Vega achieves strong performance on video generation benchmarks such as VBench and video understanding benchmarks like VideoMME.

Problem

• video understanding
• video generation
• unified framework
• semantic representation
• temporal coherence

Innovation

• unified video modeling
• autoregressive-diffusion hybrid
• shared visual-textual vocabulary
• semantic token prediction
• temporal coherence