DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter

2026-02-05

AI Summary

This work addresses the challenges of complex multimodal perception-control coupling and ineffective tactile integration in bimanual robotic dexterous manipulation by proposing a decoupled multimodal Diffusion Transformer (DiT) strategy. The approach disentangles fusion pathways for visual, action, proprioceptive, and tactile signals through a plug-in tactile adapter, leveraging self-attention, cross-attention, adaptive layer normalization, and lightweight LoRA fine-tuning. To support this research, the authors introduce DECO-50, a large-scale bimanual manipulation dataset comprising four scenarios, 28 subtasks, over 50 hours of video (∼5 million frames), and 8,000 successful trajectories. Experimental results demonstrate that the proposed method significantly enhances both task performance and generalization capability in bimanual robotic manipulation.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Overview of the Proposed DECO Framework.} DECO is a DiT-based policy that decouples multimodal conditioning. Image and action tokens interact via joint self attention, while proprioceptive states and optional conditions are injected through adaptive layer normalization. Tactile signals are injected via cross attention, while a lightweight LoRA-based adapter is used to efficiently fine-tune the pretrained policy. DECO is also accompanied by DECO-50, a bimanual dexterous manipulation dataset with tactile sensing, consisting of 4 scenarios and 28 sub-tasks, covering more than 50 hours of data, approximately 5 million frames, and 8,000 successful trajectories.

Problem

• bimanual dexterous manipulation
• multimodal fusion
• tactile sensing
• policy learning
• robotic manipulation

Innovation

• Decoupled Multimodal Diffusion
• Tactile Adapter
• Bimanual Dexterous Manipulation
• LoRA-based Fine-tuning
• Diffusion Transformer