TouchAnything: Diffusion-Guided 3D Reconstruction from Sparse Robot Touches

2026-04-10

AI Summary

This work addresses the severely underconstrained problem of reconstructing complete and accurate 3D object geometry from sparse tactile data in visually unreliable scenarios. The authors propose a novel approach that, for the first time, transfers geometric and semantic priors from a pretrained 2D vision diffusion model into the tactile domain. By leveraging only a few tactile contact points and coarse object category information, the method optimizes a 3D shape that satisfies the tactile constraints while remaining consistent with the diffusion prior. Notably, it requires no category-specific training or learned tactile diffusion models, enabling zero-shot 3D reconstruction in open-world settings. Experiments demonstrate that the approach achieves high-fidelity geometry reconstruction from minimal tactile input, outperforming existing methods across multiple benchmarks and generalizing successfully to unseen object instances.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Accurate object geometry estimation is essential for many downstream tasks, including robotic manipulation and physical interaction. Although vision is the dominant modality for shape perception, it becomes unreliable under occlusions or challenging lighting conditions. In such scenarios, tactile sensing provides direct geometric information through physical contact. However, reconstructing global 3D geometry from sparse local touches alone is fundamentally underconstrained. We present TouchAnything, a framework that leverages a pretrained large-scale 2D vision diffusion model as a semantic and geometric prior for 3D reconstruction from sparse tactile measurements. Unlike prior work that trains category-specific reconstruction networks or learns diffusion models directly from tactile data, we transfer the geometric knowledge encoded in pretrained visual diffusion models to the tactile domain. Given sparse contact constraints and a coarse class-level description of the object, we formulate reconstruction as an optimization problem that enforces tactile consistency while guiding solutions toward shapes consistent with the diffusion prior. Our method reconstructs accurate geometries from only a few touches, outperforms existing baselines, and enables open-world 3D reconstruction of previously unseen object instances. Our project page is https://grange007.github.io/touchanything .

Problem

• 3D reconstruction
• tactile sensing
• sparse touches
• shape estimation
• occlusion

Innovation

• diffusion prior
• tactile sensing
• 3D reconstruction
• sparse touches
• vision-to-touch transfer