Learning thin deformable object manipulation with a multi-sensory integrated soft hand

2024-11-21

arXiv.org

AI Summary

Robust and adaptive manipulation of thin, deformable objects—such as fabrics and sheet music—remains challenging due to their high compliance and complex dynamics. Method: This paper proposes an end-to-end learning framework integrating a passively compliant soft robotic hand with multimodal sensing (tactile, 6-DoF force/torque, and RGB-D). It introduces a novel “passive compliance + dual-loop hierarchical reinforcement learning” paradigm, employing a model-free hierarchical PPO algorithm that learns manipulation policies directly from raw sensor data—without requiring object property modeling or hand-tuned parameters. Contribution/Results: Implemented on a real robotic platform equipped with an underactuated soft hand and distributed sensors, the framework successfully accomplishes dexterous tasks including fabric unfolding and autonomous page-turning. Experiments demonstrate substantial improvements over state-of-the-art methods, achieving—for the first time in real-world settings—salesperson-style fabric presentation and violin sheet-music page-turning, thereby validating strong generalization capability and practical applicability.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Multimodal Learning
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: The sharing economy

Abstract

Robotic manipulation has made significant advancements, with systems demonstrating high precision and repeatability. However, this remarkable precision often fails to translate into efficient manipulation of thin deformable objects. Current robotic systems lack imprecise dexterity, the ability to perform dexterous manipulation through robust and adaptive behaviors that do not rely on precise control. This paper explores the singulation and grasping of thin, deformable objects. Here, we propose a novel solution that incorporates passive compliance, touch, and proprioception into thin, deformable object manipulation. Our system employs a soft, underactuated hand that provides passive compliance, facilitating adaptive and gentle interactions to dexterously manipulate deformable objects without requiring precise control. The tactile and force/torque sensors equipped on the hand, along with a depth camera, gather sensory data required for manipulation via the proposed slip module. The manipulation policies are learned directly from raw sensory data via model-free reinforcement learning, bypassing explicit environmental and object modeling. We implement a hierarchical double-loop learning process to enhance learning efficiency by decoupling the action space. Our method was deployed on real-world robots and trained in a self-supervised manner. The resulting policy was tested on a variety of challenging tasks that were beyond the capabilities of prior studies, ranging from displaying suit fabric like a salesperson to turning pages of sheet music for violinists.

Problem

• Robotic manipulation of thin deformable objects lacks precision
• Integration of passive compliance and multi-sensory data for adaptive grasping
• Learning manipulation policies via model-free reinforcement learning from raw data

Innovation

• Soft underactuated hand with passive compliance
• Multi-sensory integration via tactile and depth sensors
• Model-free reinforcement learning for policy training