PneuGelSight: Soft Robotic Vision-Based Proprioception and Tactile Sensing

2025-08-25

AI Summary

Soft pneumatic robots lack high-resolution, robust, and easily integrable tactile and proprioceptive sensing—hindering real-world deployment. To address this, we propose a novel vision-based perception paradigm leveraging embedded micro-cameras to reconstruct internal deformation of pneumatic soft arms with unprecedented accuracy, enabling simultaneous proprioceptive pose estimation and external contact detection. Our approach integrates optical-dynamic co-simulation, an end-to-end simulation-to-reality (Sim2Real) transfer pipeline, and an image-driven deformation decoding network—achieving zero-shot Sim2Real adaptation without any real-world calibration data. Experimental results demonstrate sub-millimeter pose estimation accuracy and reliable tactile responsiveness during complex manipulation tasks. The system significantly enhances perceptual robustness and deployment efficiency of soft robots in industrial assembly and human–robot interaction scenarios.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Low Level & Physics-based Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Soft pneumatic robot manipulators are popular in industrial and human-interactive applications due to their compliance and flexibility. However, deploying them in real-world scenarios requires advanced sensing for tactile feedback and proprioception. Our work presents a novel vision-based approach for sensorizing soft robots. We demonstrate our approach on PneuGelSight, a pioneering pneumatic manipulator featuring high-resolution proprioception and tactile sensing via an embedded camera. To optimize the sensor's performance, we introduce a comprehensive pipeline that accurately simulates its optical and dynamic properties, facilitating a zero-shot knowledge transition from simulation to real-world applications. PneuGelSight and our sim-to-real pipeline provide a novel, easily implementable, and robust sensing methodology for soft robots, paving the way for the development of more advanced soft robots with enhanced sensory capabilities.

Problem

• Enabling vision-based proprioception and tactile sensing for soft robots
• Developing a sim-to-real pipeline for sensor optimization
• Providing robust sensory capabilities for pneumatic manipulators

Innovation

• Vision-based soft robot sensing with embedded camera
• Sim-to-real pipeline for optical simulation
• Zero-shot knowledge transfer to real applications