ShapeGrasp: Simultaneous Visuo-Haptic Shape Completion and Grasping for Improved Robot Manipulation

2026-05-04

AI Summary

This work addresses the low success rate of grasping unknown objects from a single viewpoint, which stems from incomplete shape information and the absence of tactile feedback. To overcome this challenge, the authors propose an iterative framework that integrates vision and touch for simultaneous grasping and shape completion. Starting from a single RGB-D image, an implicit 3D representation is initialized and used within a physics simulator to generate an initial grasp. After each grasp attempt, contact points and occupancy information are fused to dynamically refine the object’s shape estimate, enabling closed-loop replanning of subsequent grasps. The system achieves, for the first time on a real robotic platform, online grasp-driven 3D reconstruction with multimodal fusion. Experiments on two robotic arms demonstrate grasp success rates of 91% and 84% for two-finger and three-finger grippers, respectively, while significantly improving reconstruction accuracy.

Abstract

Humans grasp unfamiliar objects by combining an initial visual estimate with tactile and proprioceptive feedback during interaction. We present ShapeGrasp, a robotic implementation of this approach. The proposed method is an iterative grasp-and-complete pipeline that couples implicit surface visuo-haptic shape completion (creation of full 3D shape from partial information) with physics-based grasp planning. From a single RGB-D view, ShapeGrasp infers a complete shape (point cloud or triangular mesh), generates candidate grasps via rigid-body simulation, and executes the best feasible grasp. Each grasp attempt yields additional geometric constraints -- tactile surface contacts and space occupied by the gripper body -- which are fused to update the object shape. Failures trigger pose re-estimation and regrasping using the refined shape. We evaluate ShapeGrasp in the real world using two different robots and grippers. To the best of our knowledge, this is the first approach that updates shape representations following a real-world grasp. We achieved superior results over baselines for both grippers (grasp success rate of 84% with a three-finger gripper and 91% with a two-finger gripper), while improving the 3D shape reconstruction quality in all evaluation metrics used.

Problem

• visuo-haptic
• shape completion
• robotic grasping
• 3D reconstruction
• tactile feedback

Innovation

• visuo-haptic shape completion
• iterative grasp-and-complete pipeline
• physics-based grasp planning
• implicit surface reconstruction
• real-world robotic grasping