PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop

2025-03-12

AI Summary

Current large-scale video generation models produce visually realistic outputs but exhibit severe physical inaccuracies—e.g., violating Newtonian mechanics in free-fall motion—rendering them unreliable as world simulators. To address this, we propose a physics-driven post-training framework that leverages a novel reward model trained on small-scale physics simulations to guide diffusion models toward physically consistent dynamics. We further introduce the first diagnostic benchmark explicitly designed for evaluating physical consistency in video generation. Our method integrates reinforcement learning–inspired reward modeling, distribution alignment analysis, and physics-informed fine-tuning. Experiments demonstrate that our framework significantly improves physical fidelity of mainstream models on free-fall tasks—reducing trajectory error by up to 62%. Moreover, we systematically uncover fundamental limitations of post-training: poor generalization across diverse physical scenarios and inability to faithfully capture real-world dynamical distributions—constituting the first such characterization in the literature.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Large-scale pre-trained video generation models excel in content creation but are not reliable as physically accurate world simulators out of the box. This work studies the process of post-training these models for accurate world modeling through the lens of the simple, yet fundamental, physics task of modeling object freefall. We show state-of-the-art video generation models struggle with this basic task, despite their visually impressive outputs. To remedy this problem, we find that fine-tuning on a relatively small amount of simulated videos is effective in inducing the dropping behavior in the model, and we can further improve results through a novel reward modeling procedure we introduce. Our study also reveals key limitations of post-training in generalization and distribution modeling. Additionally, we release a benchmark for this task that may serve as a useful diagnostic tool for tracking physical accuracy in large-scale video generative model development.

Problem

• Improving physical accuracy in video generation models.
• Fine-tuning models for accurate object freefall simulation.
• Addressing generalization and distribution modeling limitations.

Innovation

• Fine-tuning with simulated videos for accurate physics
• Novel reward modeling to enhance dropping behavior
• Benchmark for tracking physical accuracy in video models