3D Point World Models: Point Completion Enables More Accurate Dynamics Learning

2026-06-30

AI Summary

Existing video- or partial point cloud–based dynamics models suffer from geometric inconsistency, occlusion sensitivity, and error accumulation over long-horizon predictions, limiting their reliability for planning. This work proposes a task-agnostic, purely 3D world model that tightly integrates point cloud completion with action-conditioned dynamics modeling: it first completes observed partial point clouds into geometrically coherent full scenes and then learns action-driven dynamic evolution directly in the completed 3D space. The approach achieves geometrically consistent and robust long-horizon prediction, enabling stable trajectory forecasting over 100–300+ steps across diverse robotic platforms and tabletop manipulation tasks. It further demonstrates successful sim-to-real transfer, facilitates efficient model-predictive control, and exhibits rapid adaptability to novel tasks.

Abstract

Learning predictive models of the world enables robotic control through planning, potentially allowing robots to improvise solutions on new tasks. However, large video-based dynamics models lack explicit 3D spatial structure and suffer from geometrically inconsistent long-term rollouts with compounding errors. Emerging 3D dynamics models based on partial point clouds improve geometric consistency but remain sensitive to occlusions and accumulated prediction drift. To address these challenges, we present 3D Point World Models (3DPWM) - a task-agnostic world model that operates entirely in 3D space by first completing partial point clouds and then learning action-conditioned dynamics in this completed 3D scene. By operating on completed geometry, 3DPWM enables reliable long-horizon rollouts and more accurate cost evaluation for model-based planning while supporting adaptation to new tasks. Experiments across different robotic embodiments and tabletop manipulation benchmarks demonstrate that 3DPWM achieves significantly more reliable long-horizon rollouts (100-300+ steps), supports both open-loop and closed-loop planning, and enables successful sim-to-real transfer.

Problem

• 3D dynamics models
• point cloud completion
• occlusion sensitivity
• prediction drift
• long-horizon rollouts

Innovation

• 3D point cloud completion
• world models
• dynamics learning
• model-based planning
• sim-to-real transfer