Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach

2025-02-05

AI Summary

Existing video generation models suffer from non-physical deformations, object hallucination, and motion discontinuities due to the absence of explicit 3D geometric and physical constraints—especially in solid-body interaction tasks (e.g., grasping, collision). To address this, we propose the first 3D-aware video generation framework: (1) we introduce PointVid, the first task-oriented video dataset with precise 3D point-trajectory annotations; (2) we design a pixel-aligned 3D point-trajectory regularization mechanism that explicitly enforces geometric integrity and physically plausible motion dynamics; and (3) we fine-tune a latent diffusion model to enable 3D-coordinate-aware generation. Our approach is the first to systematically mitigate deformation artifacts and structural collapse in contact-rich scenarios. Quantitatively and qualitatively, it significantly improves 3D structural consistency, motion smoothness, and physical plausibility in RGB video generation.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We present a novel video generation framework that integrates 3-dimensional geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D point trajectories and align them in pixel space. The resulting 3D-aware video dataset, PointVid, is then used to fine-tune a latent diffusion model, enabling it to track 2D objects with 3D Cartesian coordinates. Building on this, we regularize the shape and motion of objects in the video to eliminate undesired artifacts, eg, nonphysical deformation. Consequently, we enhance the quality of generated RGB videos and alleviate common issues like object morphing, which are prevalent in current video models due to a lack of shape awareness. With our 3D augmentation and regularization, our model is capable of handling contact-rich scenarios such as task-oriented videos. These videos involve complex interactions of solids, where 3D information is essential for perceiving deformation and contact. Furthermore, our model improves the overall quality of video generation by promoting the 3D consistency of moving objects and reducing abrupt changes in shape and motion.

Problem

• Enhance video quality via 3D geometry integration
• Eliminate nonphysical deformation artifacts in videos
• Improve 3D consistency in dynamic object interactions

Innovation

• 3D point trajectories integration
• Latent diffusion model fine-tuning
• 3D consistency regularization