RoboHitch: Learning Visual Affordance from Disordered Keypoints for Hitch Knots Tying

2026-05-23

AI Summary

This work addresses the challenge of robotic knot-tying failures caused by self-occlusion and difficulties in tracking topological states. The authors propose a vision-based manipulation strategy that operates without requiring explicit topological ordering or ordered keypoints. Their approach integrates a dynamic graph autoencoder and a convolutional autoencoder to extract geometric and visual features of the rope, respectively, and employs a bidirectional cross-attention mechanism to fuse these multimodal representations for predicting grasp-and-place actions. Notably, this method achieves robust hitch knot tying using unordered 3D keypoints and RGB images as input, demonstrating strong generalization and high success rates across diverse real-world scenarios with significant self-occlusion.

Abstract

Robotic manipulation of deformable linear objects (DLOs) presents significant challenges due to complex dynamics and frequent self-occlusions. Existing robotic knot tying methods typically rely on precise topological state tracking with ordered keypoints and explicit edge connectivity. This reliance makes them prone to failures due to tracking drift and topology mismatch caused by repeated bending and crossings during knot formation.To address these limitations, we introduce RoboHitch, a novel framework that learns to perform hitch knot tying from human demonstrations using only disordered 3D keypoints and RGB images. This eliminates the need for explicit topological order, allowing for more flexible manipulation. Our method employs a dynamic Graph Autoencoder to extract geometric features from untracked keypoints, complemented by a Convolutional Autoencoder that captures essential visual context. A bidirectional cross-attention mechanism then fuses these modalities to jointly predict pick and place affordances, facilitating implicit reasoning about the rope's state and enabling knot tying under occlusion.Real-world experiments demonstrate the effectiveness and generalizability of our approach, successfully completing hitch knots in scenarios with self-occlusions.

Problem

• deformable linear objects
• knot tying
• topological state tracking
• self-occlusions
• keypoint ordering

Innovation

• disordered keypoints
• visual affordance
• deformable linear objects
• graph autoencoder
• cross-attention