TacGen: Touch Is a Necessary Dimension of Physical-World Representation -- Addressing Tactile Data Scarcity with Scalable Vision-to-Touch Alignment and Generation

2026-06-27

AI Summary

This work addresses the scarcity of tactile data, which hinders effective representation learning of contact-dependent physical properties such as hardness, density, and mass. The authors propose the first vision-to-touch generative alignment framework, leveraging a DINOv2 backbone to extract visual features and integrating vision–tactile contrastive learning with a residual MLP-based latent generator to synthesize high-fidelity tactile latents on the TACTO simulation platform. Experiments demonstrate that the method substantially outperforms purely visual models, achieving marked improvements in both attribute prediction and manipulation tasks—boosting success rates from 0.246 to 0.979—and producing synthetic tactile signals nearly indistinguishable from real ones (similarity scores of +0.589 vs. +0.585). This study provides the first empirical validation of tactile sensing’s necessity for learning contact-dependent physical representations and reveals a notable gap between generative fidelity and representational utility.

Abstract

Touch resolves the physical-property ambiguity left by vision: exploratory contact recovers shape, texture, compliance, and material, and visuo-haptic object representations converge in ventral visual cortex. We ask whether representation learning can reproduce this grounding. TacGen mitigates the tactile-data scarcity bottleneck by combining pre-specified V+T contrastive alignment with a latent-space residual-MLP V->T generator that synthesizes tactile latents from RGB for tactile-data scaling. With matched DINOv2 backbones, splits, and probes, V+T improves matched V-only on mass (Delta R^2=+0.570), density (Delta acc=+0.067), hardness (+0.117), and uncertainty-banded force labels (Delta R^2=+0.281); all CIs exclude zero. The same representation lifts matched-capacity TACTO manipulation 0.246->0.979 while V-only capacity scaling accounts for only 4.5% of the gap, preserving 95.5%. The generator reaches cross-seed +0.589, with real tactile +0.585 inside the seed interval; the architecture comparison shows a 13pp downstream gap between reconstruction quality and representation utility. Across five-seed SSVTP/TVL reproductions, YCB-Sight transfer, three-backbone checks, permutation/random-feature controls, hash-verified manifests, and measured-force validation checks, the evidence supports the claim that touch supplies a necessary physical evidence channel for representations of contact-dependent properties.

Problem

• tactile data scarcity
• physical-world representation
• touch
• vision-to-touch alignment
• contact-dependent properties

Innovation

• vision-to-touch generation
• tactile data scarcity
• visuo-haptic alignment
• latent-space synthesis
• physical-world representation