Geometry-Aware Motion Latents for Learning Robust Manipulation Policies

2026-07-06

AI Summary

This work proposes a novel approach to robotic manipulation that moves beyond reliance on visual appearance patterns, which often fail to capture the underlying physical dynamics of motion. Instead, it models the evolution of point clouds in four-dimensional spatio-temporal space to learn discrete, geometry-aware motion latent variables that encode three-dimensional geometric transformations rather than pixel-level appearances. By focusing latent motion representation learning explicitly on predicting geometric changes, the method achieves vision-context-invariant and physically consistent action abstraction. Requiring only single-view RGB-D input and a small number of demonstrations, the approach demonstrates robust manipulation performance in cluttered environments and achieves state-of-the-art results across multiple benchmarks, underscoring the critical role of geometric understanding in robotic manipulation tasks.

Abstract

Learning motion latents for robotic manipulation heavily relies on extracting motion patterns from visual sequences, yet effective action abstractions require understanding three-dimensional geometric transformations. Here, we introduce GeoMoLa (Geometry-Aware Motion Latents), which learns discrete motion latent codes by predicting how point clouds evolve during manipulation rather than reconstructing visual observations. This four-dimensional objective -- spatial geometry changing through time -- forces latent representations to encode actual physical motion rather than appearance patterns. GeoMoLa achieves state-of-the-art performance using only single-view RGB-D input, while existing methods require multi-view reconstruction, succeeding across diverse manipulation benchmarks. Our ablations reveal that geometric prediction is the key to driving performance, quantitatively validating that manipulation depends on spatial understanding. Furthermore, the learned codes exhibit effective motion abstraction: applying them to novel scenes produces physically consistent transformations regardless of visual context. Our real-world experiments also confirm this robustness capability, achieving robust manipulation with minimal demonstrations in cluttered environments where geometric reasoning determines success. Thus, we demonstrate that effective motion latents for robot control can better emerge from understanding motion through its three-dimensional effects rather than pixel-level patterns.

Problem

• motion latents
• geometric reasoning
• robotic manipulation
• 3D motion understanding
• visual robustness

Innovation

• geometry-aware
• motion latents
• point cloud dynamics
• robotic manipulation
• 3D geometric reasoning