Structural Action Transformer for 3D Dexterous Manipulation

2026-03-04

AI Summary

This work addresses the challenge of transferring dexterous manipulation skills across heterogeneous robotic hands in imitation learning, where reliance on 2D observations and temporally ordered action representations hinders generalization. To overcome this limitation, the authors propose the Structured Action Transformer (SAT), which takes 3D point clouds as input and models actions as unordered, variable-length sets of joint trajectories. SAT introduces an embodied joint codebook that encodes functional and kinematic properties of joints, and employs a continuous-time flow matching objective for policy learning. By departing from conventional sequential action modeling and leveraging the Transformer architecture with large-scale pretraining on heterogeneous data, SAT significantly improves sample efficiency and cross-embodiment skill transfer in both simulated and real-world tasks.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Achieving human-level dexterity in robots via imitation learning from heterogeneous datasets is hindered by the challenge of cross-embodiment skill transfer, particularly for high-DoF robotic hands. Existing methods, often relying on 2D observations and temporal-centric action representation, struggle to capture 3D spatial relations and fail to handle embodiment heterogeneity. This paper proposes the Structural Action Transformer (SAT), a new 3D dexterous manipulation policy that challenges this paradigm by introducing a structural-centric perspective. We reframe each action chunk not as a temporal sequence, but as a variable-length, unordered sequence of joint-wise trajectories. This structural formulation allows a Transformer to natively handle heterogeneous embodiments, treating the joint count as a variable sequence length. To encode structural priors and resolve ambiguity, we introduce an Embodied Joint Codebook that embeds each joint's functional role and kinematic properties. Our model learns to generate these trajectories from 3D point clouds via a continuous-time flow matching objective. We validate our approach by pre-training on large-scale heterogeneous datasets and fine-tuning on simulation and real-world dexterous manipulation tasks. Our method consistently outperforms all baselines, demonstrating superior sample efficiency and effective cross-embodiment skill transfer. This structural-centric representation offers a new path toward scaling policies for high-DoF, heterogeneous manipulators.

Problem

• dexterous manipulation
• cross-embodiment transfer
• 3D spatial reasoning
• heterogeneous robotic hands
• imitation learning

Innovation

• Structural Action Transformer
• 3D dexterous manipulation
• cross-embodiment transfer
• Embodied Joint Codebook
• flow matching