VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models

2025-05-29

AI Summary

Current text-to-video (T2V) diffusion models generate high-fidelity videos but suffer from poor physical plausibility due to insufficient modeling of physical laws. To address this, we propose the first physics-knowledge-injection framework for fine-tuning T2V diffusion models, which distills physical understanding from a video self-supervised foundation model into the diffusion generator to enhance physical consistency. Our core innovation is a token-level relation distillation (TRD) loss that enables soft spatiotemporal alignment between the teacher and student models. Evaluated on CogVideoX, our method significantly improves performance on physical commonsense benchmarks—yielding videos that better adhere to fundamental physical principles such as gravity, collision dynamics, and motion continuity. This work establishes a new paradigm for controllable, physics-aware video generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advancements in text-to-video (T2V) diffusion models have enabled high-fidelity and realistic video synthesis. However, current T2V models often struggle to generate physically plausible content due to their limited inherent ability to accurately understand physics. We found that while the representations within T2V models possess some capacity for physics understanding, they lag significantly behind those from recent video self-supervised learning methods. To this end, we propose a novel framework called VideoREPA, which distills physics understanding capability from video understanding foundation models into T2V models by aligning token-level relations. This closes the physics understanding gap and enable more physics-plausible generation. Specifically, we introduce the Token Relation Distillation (TRD) loss, leveraging spatio-temporal alignment to provide soft guidance suitable for finetuning powerful pre-trained T2V models, a critical departure from prior representation alignment (REPA) methods. To our knowledge, VideoREPA is the first REPA method designed for finetuning T2V models and specifically for injecting physical knowledge. Empirical evaluations show that VideoREPA substantially enhances the physics commonsense of baseline method, CogVideoX, achieving significant improvement on relevant benchmarks and demonstrating a strong capacity for generating videos consistent with intuitive physics. More video results are available at https://videorepa.github.io/.

Problem

• Improving physics plausibility in text-to-video generation
• Aligning token relations to distill physics understanding
• Enhancing video generation with intuitive physics consistency

Innovation

• Distills physics from foundation models
• Uses Token Relation Distillation loss
• Aligns token-level relations for physics