Generative Physical AI in Vision: A Survey

2025-01-19

AI Summary

Current generative AI models produce high-fidelity 3D/4D visual content but generally lack explicit modeling and adherence to physical laws, limiting their deployment in physics-critical applications such as robotics and autonomous driving. This paper formally defines the emerging field of *physics-aware generative visual AI*, proposing a dual-path taxonomy—explicit physics simulation versus implicit physics learning—and introducing the first multi-dimensional evaluation framework targeting physical consistency. Methodologically, it integrates physics engines (PyBullet/PhysX), neural simulation, physics-constrained loss functions, differentiable rendering, and multimodal physical priors. The work surveys over 100 state-of-the-art studies and open-sources an authoritative literature repository (GitHub). Key contributions include: (1) establishing the theoretical foundations and boundaries of the field; (2) providing a systematic technical roadmap and standardized benchmark suite; and (3) identifying four pivotal future directions—differentiable physics modeling, embodied generative AI, causal physical representation learning, and closed-loop world simulation.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Embodied AI

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Generative Artificial Intelligence (AI) has rapidly advanced the field of computer vision by enabling machines to create and interpret visual data with unprecedented sophistication. This transformation builds upon a foundation of generative models to produce realistic images, videos, and 3D or 4D content. Traditionally, generative models primarily focus on visual fidelity while often neglecting the physical plausibility of generated content. This gap limits their effectiveness in applications requiring adherence to real-world physical laws, such as robotics, autonomous systems, and scientific simulations. As generative AI evolves to increasingly integrate physical realism and dynamic simulation, its potential to function as a"world simulator"expands-enabling the modeling of interactions governed by physics and bridging the divide between virtual and physical realities. This survey systematically reviews this emerging field of physics-aware generative AI in computer vision, categorizing methods based on how they incorporate physical knowledge-either through explicit simulation or implicit learning. We analyze key paradigms, discuss evaluation protocols, and identify future research directions. By offering a comprehensive overview, this survey aims to help future developments in physically grounded generation for vision. The reviewed papers are summarized at https://github.com/BestJunYu/Awesome-Physics-aware-Generation.

Problem

• Generative AI
• Physical Realism
• Reality Simulation

Innovation

• Generative AI
• Physical Rule Learning
• Virtual-Reality Integration