SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects

2026-07-05

AI Summary

Existing benchmarks for deformable object manipulation primarily focus on task success rates while neglecting physical safety, such as object slippage or excessive deformation. This work proposes SoftVTBench, the first safety-aware visuo-tactile benchmark, which simulates deformable objects in Isaac Sim using finite element methods and integrates multi-view RGB vision, marker-based RGB tactile sensing, proprioception, and language instructions. The benchmark comprises four task suites and uniquely introduces physical safety—defined as no dropping and deformation below a specified threshold—as an independent evaluation metric. Experimental results demonstrate that incorporating tactile information significantly improves safety success rates (from 21.4% to 35.6%) and reduces deformation while maintaining comparable task success rates, revealing that evaluating solely on task completion substantially overestimates policy performance.

Abstract

Deformable object manipulation poses challenges beyond task completion: successful execution must also maintain safe physical interaction, holding the object stably without slip or drop while avoiding excessive deformation. However, existing manipulation benchmarks are predominantly success-oriented and rarely evaluate whether a policy remains physically safe throughout execution. We present SoftVTBench, a safety-aware visuo-tactile benchmark for physically constrained deformable object manipulation. Built in Isaac Sim with finite-element-simulated deformable objects, SoftVTBench provides multi-view RGB observations, RGB tactile sensing with marker motion, proprioception, and language instructions, and defines four matched task suites over object type (deformable vs. rigid) and variation axis (object vs. spatial). It separately reports Goal Success and Safety Success; the latter additionally requires no drop and peak deformation below a calibrated object-specific threshold, measured from policy-hidden privileged Finite Element Method (FEM) states. We implement pi0.5-based baselines under this protocol. Experiments show that success-only evaluation substantially overstates policy performance, as a large fraction of goal-completing rollouts still violate physical safety. Furthermore, incorporating tactile sensing improves Safety Success (e.g., from 21.4% to 35.6% on object-centric deformable tasks) and reduces object deformation during execution, while maintaining comparable Goal Success. SoftVTBench provides a reproducible benchmark for studying visuo-tactile deformable manipulation under physical interaction constraints.

Problem

• deformable object manipulation
• physical safety
• robotic manipulation
• safety-aware benchmark
• visuo-tactile sensing

Innovation

• safety-aware manipulation
• visuo-tactile benchmark
• deformable object manipulation
• finite element method
• physical interaction constraints