Feeling the Unexpected: ResTacVLA for Contact-Rich Manipulation via Residual Tactile Representation

2026-07-03

AI Summary

This work addresses the challenge of modality collapse in vision-language-action models, where high-bandwidth visual signals often overshadow sparse tactile cues, impairing performance on contact-intensive manipulation tasks. Inspired by predictive coding, the authors propose a residual tactile representation that captures discrepancies between visual priors and actual tactile observations. A vector-quantized bottleneck is employed to extract latent contact primitives from these residuals. Additionally, an adaptive gating mechanism, conditioned on visual uncertainty, prioritizes the integration of tactile residuals when visual inputs are unreliable. This approach effectively mitigates multimodal bandwidth mismatch by emulating neural “surprise” signals for dynamic tactile integration. The method significantly outperforms existing approaches across diverse contact-rich tasks and demonstrates robustness to dynamic perturbations.

Abstract

Tactile perception is indispensable for contact-rich manipulation, yet integrating it into Vision-Language-Action (VLA) models often induces modality collapse, where high-bandwidth visual features overshadow sparse tactile cues. Inspired by Predictive Coding, a neural mechanism where the brain attenuates predictable inputs to prioritize surprising stimuli, we propose ResTacVLA. Rather than treating tactile data as raw input, we reformulate it as a Residual Tactile Representation capturing the discrepancy between visual priors and physical sensations. By filtering out visually predictable dynamics, this formulation transforms sparse tactile signals into dense, high-value information gain, thereby inherently resolving the bandwidth mismatch. These residuals are discretized through a Vector Quantized (VQ) bottleneck into Latent Contact Primitives that capture critical events missed by vision. Analogous to the neural surprise signal, we leverage the uncertainty of the visual prior to adaptively gate tactile integration, prioritizing residuals specifically during visually unreliable phases to explicitly prevent visual dominance. Experimental results show that ResTacVLA consistently outperforms all baselines on a diverse set of contact-rich manipulation tasks, while remaining robust to unexpected dynamic disturbances. Project page: https://awilekong.github.io/ResTacVLA-Website/

Problem

• tactile perception
• modality collapse
• contact-rich manipulation
• Vision-Language-Action models
• sensory integration

Innovation

• Residual Tactile Representation
• Predictive Coding
• Vector Quantization
• Latent Contact Primitives
• Modality Collapse