DexFormer: Cross-Embodied Dexterous Manipulation via History-Conditioned Transformer

2026-02-09

AI Summary

This work addresses the challenge of policy generalization across diverse dexterous manipulators stemming from their heterogeneous morphologies and dynamics. To overcome this, the authors propose an end-to-end, perception-conditioned cross-embodiment control strategy based on a Transformer architecture conditioned on historical observations. At runtime, the model infers the hand’s morphology and dynamical properties, enabling unified control of multiple heterogeneous dexterous hands. By training end-to-end on procedurally generated, diverse hand assets, the approach achieves zero-shot transfer to previously unseen hands—such as the Leap Hand, Allegro Hand, and Rapid Hand—within a single policy. This eliminates the need for hand-specific training or reliance on a shared action space, significantly enhancing both generalization and scalability of the learned policy.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Dexterous manipulation remains one of the most challenging problems in robotics, requiring coherent control of high-DoF hands and arms under complex, contact-rich dynamics. A major barrier is embodiment variability: different dexterous hands exhibit distinct kinematics and dynamics, forcing prior methods to train separate policies or rely on shared action spaces with per-embodiment decoder heads. We present DexFormer, an end-to-end, dynamics-aware cross-embodiment policy built on a modified transformer backbone that conditions on historical observations. By using temporal context to infer morphology and dynamics on the fly, DexFormer adapts to diverse hand configurations and produces embodiment-appropriate control actions. Trained over a variety of procedurally generated dexterous-hand assets, DexFormer acquires a generalizable manipulation prior and exhibits strong zero-shot transfer to Leap Hand, Allegro Hand, and Rapid Hand. Our results show that a single policy can generalize across heterogeneous hand embodiments, establishing a scalable foundation for cross-embodiment dexterous manipulation. Project website: https://davidlxu.github.io/DexFormer-web/.

Problem

• dexterous manipulation
• embodiment variability
• cross-embodiment
• robotic hands
• generalization

Innovation

• cross-embodiment
• dexterous manipulation
• history-conditioned transformer
• zero-shot transfer
• morphology inference