Pix2Act: Image-Space Manipulation Policies with Equivariant Augmentation

2026-07-13

AI Summary

This work addresses the challenges of trajectory clipping and insufficient precision when mapping 3D manipulation policies to 2D image-space trajectories by introducing a novel paradigm based on keypoint trajectory prediction in image space. The method generates continuous 2D keypoint trajectories across multi-view camera planes and recovers the end-effector’s full 6D pose losslessly via triangulation, thereby reducing high-dimensional 3D control to a more learnable 2D task. Key innovations include an equivariant data augmentation mechanism that jointly transforms observations and actions, and a rotation-aware multi-view fusion network that implicitly expands the support of the data distribution while preserving geometric consistency across views. Experiments demonstrate that the proposed approach significantly outperforms existing baselines across diverse simulated and real-world manipulation tasks and exhibits strong robustness under camera perturbations.

Abstract

Representing manipulation actions as 2D trajectories in the camera plane provides a compact and interpretable basis for learning complex 3D manipulation policies. However, it also creates challenges from out-of-frame trajectories and limited precision. We propose Pix2Act, an imitation learning method that addresses these challenges by generating continuous image-space keypoint trajectories in each camera plane and losslessly recovering end-effector poses via triangulation. This reformulates high-dimensional 3D control as a simpler, more learnable 2D prediction problem. Crucially, it aligns observations and actions in the same coordinate space, enabling equivariant transformations to jointly rotate individual camera images together with their image-space actions. We analyze the symmetry properties of this augmentation and design a network architecture that can fuse multiple camera views while respecting their per-view rotations. As a result, Pix2Act implicitly enlarges the support of the data distribution and learns invariant action structures across transformations, yielding improved generalization and overall performance. Across diverse simulated and real-world manipulation tasks, Pix2Act outperforms state-of-the-art baselines and remains robust under camera perturbations.

Problem

• image-space manipulation
• 3D manipulation policies
• equivariant augmentation
• imitation learning
• camera perturbations

Innovation

• equivariant augmentation
• image-space trajectories
• triangulation-based pose recovery
• multi-view fusion
• imitation learning