Learning Force Distribution Estimation for the GelSight Mini Optical Tactile Sensor Based on Finite Element Analysis

2024-10-08

arXiv.org

AI Summary

GelSight Mini optical tactile sensors struggle to accurately reconstruct spatially resolved normal and tangential force distributions from silicone deformation images. Method: We propose a physics-guided deep learning approach that leverages high-fidelity finite element analysis (FEA) to generate pixel-level ground-truth 3D force maps—used as supervision to train a U-Net architecture for direct regression of full-contact-area force fields from raw tactile images. Contribution/Results: The method significantly improves cross-indenter and cross-sensor generalization and enables real-time inference (>30 FPS). Evaluated on a novel open-source dataset, it achieves <8.2% error in normal force prediction and <12.6% in tangential force prediction. To foster reproducibility and benchmarking, we publicly release the source code, FEA simulation models, annotated dataset, and pre-trained models—establishing a new standard for vision-based tactile force perception research.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Interaction Techniques and Devices

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Contact-rich manipulation remains a major challenge in robotics. Optical tactile sensors like GelSight Mini offer a low-cost solution for contact sensing by capturing soft-body deformations of the silicone gel. However, accurately inferring shear and normal force distributions from these gel deformations has yet to be fully addressed. In this work, we propose a machine learning approach using a U-net architecture to predict force distributions directly from the sensor's raw images. Our model, trained on force distributions inferred from Finite Element Analysis (FEA), demonstrates promising accuracy in predicting normal and shear force distributions for the commercially available GelSight Mini sensor. It also shows potential for generalization across indenters, sensors of the same type, and for enabling real-time application. The codebase, dataset and models are open-sourced and available at https://feats-ai.github.io .

Problem

• Accurately inferring shear and normal force distributions from gel deformations.
• Developing a machine learning model for real-time force distribution prediction.
• Generalizing force distribution estimation across different indenters and sensors.

Innovation

• U-net architecture predicts force distributions
• Training based on Finite Element Analysis data
• Real-time application potential demonstrated