Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation

2026-06-29

AI Summary

Existing optical tactile methods struggle to accurately discern subtle contact states due to their reliance on raw images or accumulated motion fields, often resulting in perceptual ambiguity. To address this limitation, this work proposes a dynamic tactile representation that integrates both instantaneous and cumulative motion correlations and, for the first time, explicitly incorporates dynamic priors of tactile motion to distinguish fine-grained contact differences. Building upon this representation, the authors design a unified multimodal fusion architecture based on a Mixture-of-Transformers, which effectively preserves the distinct characteristics of visual and tactile modalities while enabling rich cross-modal interactions. Evaluated on contact-intensive manipulation tasks, the proposed approach significantly outperforms existing tactile representations and fusion strategies, demonstrating enhanced sensitivity to minute contact variations and improved manipulation performance.

Abstract

Visuo-Tactile policies leveraging optical tactile sensors have shown great promise in contact-rich manipulation. These sensors achieve high spatial resolution and multi-dimensional force sensing by utilizing an internal camera to monitor the deformation of their elastic gel surface, thereby indirectly inferring tactile cues. Despite their advantages, extracting fine-grained contact states necessary for contact-rich manipulation remains an open challenge. Existing methods typically use either raw images or cumulative motion fields to represent tactile cues. However, both are prone to perception ambiguity. Raw tactile images mainly capture appearance changes, while cumulative motion fields only reflect the aggregate gel deformation. Consequently, distinct fine-grained contact states can exhibit highly similar patterns, making it difficult to explicitly distinguish subtle contact variations. To address this issue, we explore the dynamic priors of tactile motion and discover that the correlation between transient and cumulative motion can explicitly distinguish fine-grained contact states. Based on this insight, we propose a motion-aware tactile representation to facilitate contact-rich manipulation. Beyond tactile representation, effective fusion of tactile and visual modalities is also critical. Most existing fusion methods either directly concatenate features from each modality or train modality-specific networks separately and fuse their outputs. However, these strategies struggle to simultaneously model cross-modal interactions and preserve modality-specific characteristics. In this work, we take advantage of the Mixture-of-Transformers architecture and propose a unified modality-aware visuo-tactile policy that captures cross-modal complementarity while maintaining modality-specific properties.

Problem

• visuo-tactile policy
• tactile motion correlation
• contact-rich manipulation
• modality fusion
• fine-grained contact states

Innovation

• tactile motion correlation
• modality-aware fusion
• visuo-tactile policy
• Mixture-of-Transformers
• contact-rich manipulation