Gen4U: Unifying Video Generation and Understanding via Diffusion

2026-07-07

AI Summary

Traditional diffusion models struggle to effectively capture high-level semantics, limiting their applicability in video understanding. This work systematically analyzes intermediate activations of video diffusion models and reveals that their latent representations exhibit a highly structured organization across depth and noise levels. Building on this insight, the authors propose Gen4U, a framework that repurposes a frozen, large-scale video diffusion model as a universal encoder via a single forward pass. Gen4U integrates mutual kNN alignment metrics and attention mechanisms to decode spatial details, enabling unified generation and understanding without any fine-tuning. The approach achieves strong performance across diverse tasks—including video classification, depth estimation, camera pose estimation, and text-to-video captioning—while preserving the original model’s high-quality generative capabilities.

Abstract

Prior work suggests that diffusion representations capture low-level geometry but struggle with high-level semantics. We demonstrate that state-of-the-art video diffusion models overcome this limitation. By systematically probing their intermediate activations using recent mutual-kNN alignment metrics, we reveal a highly structured latent space where visual representations evolve across both network depth and noise levels. We show that while moderate noise levels yield linearly separable global semantics, fine-grained details persist at lower noise levels but become spatially scattered, requiring attention mechanisms to decode. Building on these insights, we introduce Gen4U (Generation for Understanding), a framework that repurposes these generative representations with a single forward pass. Our experiments establish that frozen, large-scale video diffusion models function as highly competitive video encoders across a wide spectrum of tasks, spanning semantic and non-semantic objectives (video classification, depth estimation, camera pose estimation, image and video captioning). Bypassing fine-tuning, Gen4U unifies the generation and understanding paradigms, achieving strong perception performance while fully preserving the model's ability to generate high-quality video.

Problem

• video diffusion models
• representation learning
• video understanding
• generation-unification
• latent space structure

Innovation

• video diffusion models
• representation probing
• mutual-kNN alignment
• Gen4U
• frozen encoder