Skill-Aware Diffusion for Generalizable Robotic Manipulation

2026-01-16

AI Summary

Existing robotic manipulation approaches typically model tasks in isolation, overlooking the shared structure across skill hierarchies and thus limiting generalization. This work proposes a skill-aware diffusion model that leverages skill-level representations to guide the generation of object-oriented motion trajectories. By integrating learnable skill tokens with a skill retrieval strategy, the method effectively maps 2D motion cues into executable 3D actions. Notably, it is the first to explicitly model skill priors and introduces IsaacSkill, a high-fidelity simulation dataset designed to facilitate sim-to-real transfer. Evaluated in both simulated and real-world environments, the approach significantly outperforms current methods across a variety of manipulation tasks, demonstrating exceptional cross-task generalization capabilities.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Robust generalization in robotic manipulation is crucial for robots to adapt flexibly to diverse environments. Existing methods usually improve generalization by scaling data and networks, but model tasks independently and overlook skill-level information. Observing that tasks within the same skill share similar motion patterns, we propose Skill-Aware Diffusion (SADiff), which explicitly incorporates skill-level information to improve generalization. SADiff learns skill-specific representations through a skill-aware encoding module with learnable skill tokens, and conditions a skill-constrained diffusion model to generate object-centric motion flow. A skill-retrieval transformation strategy further exploits skill-specific trajectory priors to refine the mapping from 2D motion flow to executable 3D actions. Furthermore, we introduce IsaacSkill, a high-fidelity dataset containing fundamental robotic skills for comprehensive evaluation and sim-to-real transfer. Experiments in simulation and real-world settings show that SADiff achieves good performance and generalization across various manipulation tasks. Code, data, and videos are available at https://sites.google.com/view/sa-diff.

Problem

• robotic manipulation
• generalization
• skill-level information
• task modeling
• motion patterns

Innovation

• Skill-Aware Diffusion
• skill-level representation
• diffusion model
• motion flow
• sim-to-real transfer