Dex4D: Task-Agnostic Point Track Policy for Sim-to-Real Dexterous Manipulation

2026-02-17

AI Summary

This work addresses the challenges of dexterous manipulation, particularly the scarcity of real-world teleoperation data and the need for task-specific environments and reward functions in simulation-based training. The authors propose a task-agnostic policy conditioned on 3D point trajectories, which learns general "any-to-any" pose manipulation skills in simulation without requiring fine-tuning for zero-shot transfer to the real world. By using object-centric point trajectories as prompts, the method flexibly composes primitive skills, significantly enhancing generalization and scalability. Integrating domain-invariant policy learning, pose estimation from video, and closed-loop online point tracking, the approach enables zero-shot deployment across diverse dexterous manipulation tasks on both simulated and real robots, outperforming existing baselines on novel objects, scenes, backgrounds, and trajectories.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Learning generalist policies capable of accomplishing a plethora of everyday tasks remains an open challenge in dexterous manipulation. In particular, collecting large-scale manipulation data via real-world teleoperation is expensive and difficult to scale. While learning in simulation provides a feasible alternative, designing multiple task-specific environments and rewards for training is similarly challenging. We propose Dex4D, a framework that instead leverages simulation for learning task-agnostic dexterous skills that can be flexibly recomposed to perform diverse real-world manipulation tasks. Specifically, Dex4D learns a domain-agnostic 3D point track conditioned policy capable of manipulating any object to any desired pose. We train this 'Anypose-to-Anypose' policy in simulation across thousands of objects with diverse pose configurations, covering a broad space of robot-object interactions that can be composed at test time. At deployment, this policy can be zero-shot transferred to real-world tasks without finetuning, simply by prompting it with desired object-centric point tracks extracted from generated videos. During execution, Dex4D uses online point tracking for closed-loop perception and control. Extensive experiments in simulation and on real robots show that our method enables zero-shot deployment for diverse dexterous manipulation tasks and yields consistent improvements over prior baselines. Furthermore, we demonstrate strong generalization to novel objects, scene layouts, backgrounds, and trajectories, highlighting the robustness and scalability of the proposed framework.

Problem

• dexterous manipulation
• task-agnostic policy
• sim-to-real transfer
• generalist policy
• zero-shot deployment

Innovation

• task-agnostic manipulation
• sim-to-real transfer
• point tracking
• dexterous manipulation
• zero-shot deployment