Shape-Space Deformer: Unified Visuo-Tactile Representations for Robotic Manipulation of Deformable Objects

2024-09-19

arXiv.org

AI Summary

Existing robotic manipulation methods for soft/deformable objects exhibit poor generalization, struggling to adapt to unseen external forces or novel objects. This paper introduces the Template-Enhanced Shape-space Deformer (TESD), the first framework unifying visual and tactile modalities into a single deformable representation space that generalizes across objects and applied forces. TESD integrates multimodal perception encoding, differentiable template-based deformation modeling, and geometry-aware regularization robust to sensor anomalies. The approach enables rapid adaptation and high-fidelity deformation reconstruction under previously unobserved forces and on unseen objects. Experiments demonstrate a 37% reduction in deformation reconstruction error under multi-force generalization scenarios, while maintaining real-time inference capability. By providing a high-fidelity, cross-object and cross-force deformable prior, TESD advances dexterous manipulation of soft materials.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Accurate modelling of object deformations is crucial for a wide range of robotic manipulation tasks, where interacting with soft or deformable objects is essential. Current methods struggle to generalise to unseen forces or adapt to new objects, limiting their utility in real-world applications. We propose Shape-Space Deformer, a unified representation for encoding a diverse range of object deformations using template augmentation to achieve robust, fine-grained reconstructions that are resilient to outliers and unwanted artefacts. Our method improves generalization to unseen forces and can rapidly adapt to novel objects, significantly outperforming existing approaches. We perform extensive experiments to test a range of force generalisation settings and evaluate our method's ability to reconstruct unseen deformations, demonstrating significant improvements in reconstruction accuracy and robustness. Our approach is suitable for real-time performance, making it ready for downstream manipulation applications.

Problem

• Accurate modeling of deformable objects for robotic manipulation
• Generalization to unseen forces and adaptation to new objects
• Real-time, robust reconstruction of diverse object deformations

Innovation

• Unified visuo-tactile representation for deformable objects
• Template augmentation for robust fine-grained reconstructions
• Real-time performance suitable for robotic manipulation