Problem
Research questions and friction points this paper is trying to address.
physically aware video generation
physical phenomena
video diffusion models
physical rule compliance
PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generation
🤖 AI Summary
Existing video generation models struggle to accurately simulate physical phenomena such as thermodynamics, mechanics, and optics, often failing to adhere to fundamental physical laws. To address this limitation, this work proposes a novel paradigm that integrates two-stage data filtering with retrieval-augmented generation (RAG). By constructing WISA-80K—a high-quality physics-aware video database—and introducing a learnable query mechanism, the method dynamically injects external physical knowledge into a video diffusion model. This approach achieves the first explicit incorporation of physical rules during the generation process, establishing state-of-the-art performance on both visual quality and physical fidelity across the PhyGenBench and VBench benchmarks, significantly outperforming existing methods.
📝 Abstract
Developing physically aware video generation models remains a significant challenge due to the difficulty in capturing diverse physical phenomena, such as thermal dynamics, mechanics, and optics. In this work, we introduce PhysRAG, a novel pipeline that enhances physical awareness in video generation through Retrieval-Augmented Generation (RAG). To address the issue of limited high-quality data, we design a two-stage data filtering pipeline based on the WISA-80K dataset, resulting in a curated set of 7K high-quality videos for training. Furthermore, we construct a physical video database and develop a mechanism to inject physical knowledge into a video diffusion model using learnable queries. Our method achieves state-of-the-art performance in both visual quality and physical rule compliance, surpassing existing models in benchmarks such as PhyGenBench and VBench. We conduct extensive ablation studies to validate the effectiveness of our key components, including the data filtering pipeline, RAG mechanism, and method for physical information extraction. To facilitate future research, our code, data, and models are prepared for release at https://github.com/sediment1024/PhysRAG.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Retrieval-Augmented Generation
Physics-Aware Video Generation
Video Diffusion Model
Physical Knowledge Injection
Data Filtering Pipeline
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
