Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models

2026-07-06

AI Summary

This work addresses the lack of large-scale real-world datasets for systematically evaluating the trade-offs between 2D pixel-space and 3D geometric approaches in modeling deformable object dynamics. To this end, the authors introduce a large-scale dataset comprising 198 everyday deformable objects and 1,980 multi-view visuo-tactile interaction sequences, captured using a surrounding camera array and a dual-arm robotic hand. By leveraging markerless visuo-tactile fusion, the dataset enables dense 3D geometry and motion tracking. It establishes the first multimodal interaction benchmark for real deformable objects, revealing fundamental trade-offs between structural priors and scalability in 2D versus 3D world models. The effectiveness of the dataset is validated through robotic manipulation tasks, laying a foundation for generalizable deformable object modeling.

Abstract

Predicting object dynamics (i.e., world modeling) is a fundamental challenge for robotic manipulation, and modeling deformable objects presents a particularly difficult case due to their high-dimensional state spaces and complex material properties. While current world models approach this through two distinct paradigms: learning the dynamics over the 2D pixel space or more explicit 3D geometric space. A systematic understanding of their relative strengths and limitations remains elusive due to the lack of diverse, large-scale real-world data. To address this, we present Deform360, a large-scale visuotactile dataset featuring 198 daily-life objects, 1,980 interaction sequences, and over 215 hours of observations from 41 surround-view cameras and bimanual tactile grippers to capture both global motion and contact-induced local deformations. Leveraging a novel markerless visuotactile 3D tracking pipeline to extract dense geometry and motion, we systematically evaluate current state-of-the-art world models, comparing 2D video models against 3D particle models. Finally, we provide a preliminary demonstration indicating the real-world applicability of our dataset by performing robot planning tasks on deformable objects. Our analysis reveals key insights into the trade-offs between structural priors and scalability, providing a solid benchmark for future research in generalizable deformable object-centric world modeling. Project website: https://deform360.lhy.xyz

Problem

• deformable objects
• world modeling
• visuotactile dataset
• object dynamics
• 3D geometric space

Innovation

• visuotactile dataset
• deformable object modeling
• 3D tracking
• world models
• multi-view perception