DexKnot: Generalizable Visuomotor Policy Learning for Dexterous Bag-Knotting Manipulation

2026-03-07

AI Summary

This work addresses the challenge of generalizing robotic knot-tying to plastic bags with diverse shapes and deformations. The authors propose a method that integrates keypoint-based manipulability with a diffusion policy. By learning shape-invariant keypoint representations from real-world manual deformation data, the approach compresses high-dimensional visual observations into sparse keypoints. A diffusion Transformer then leverages a small number of human demonstrations to generate robust action sequences. This framework significantly enhances policy generalization to unseen bag instances and complex deformations, achieving stable and high-success-rate autonomous knot-tying across a variety of novel plastic bags, thereby demonstrating its effectiveness and practicality.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Diffusion Models for Vision
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Knotting plastic bags is a common task in daily life, yet it is challenging for robots due to the bags'infinite degrees of freedom and complex physical dynamics. Existing methods often struggle in generalization to unseen bag instances or deformations. To address this, we present DexKnot, a framework that combines keypoint affordance with diffusion policy to learn a generalizable bag-knotting policy. Our approach learns a shape-agnostic representation of bags from keypoint correspondence data collected through real-world manual deformation. For an unseen bag configuration, the keypoints can be identified by matching the representation to a reference. These keypoints are then provided to a diffusion transformer, which generates robot action based on a small number of human demonstrations. DexKnot enables effective policy generalization by reducing the dimensionality of observation space into a sparse set of keypoints. Experiments show that DexKnot achieves reliable and consistent knotting performance across a variety of previously unseen instances and deformations.

Problem

• dexterous manipulation
• bag knotting
• generalization
• visuomotor policy
• deformable objects

Innovation

• keypoint affordance
• diffusion policy
• shape-agnostic representation
• visuomotor policy
• dexterous manipulation