TAP-VLA: Tactile Annotation Prompting for Vision Language Action Models

2026-06-27

AI Summary

This work addresses the challenge that existing vision-language-action models struggle to perceive contact forces due to the absence of tactile information, while naive integration of tactile signals often induces distributional shift. To overcome this, the authors propose a tactile prompting method that requires neither tactile pretraining nor architectural modifications: spatially aligned shear force fields extracted from tactile sensors are encoded as vector fields and superimposed onto multi-view RGB images, enabling interpretable injection of tactile cues while preserving the original visual pretraining distribution. By integrating vision-tactile alignment, multi-view fusion, and model fine-tuning, the approach achieves a 78% success rate across four high-contact manipulation tasks—significantly outperforming purely visual fine-tuning (<50%) and other tactile fusion baselines.

Abstract

Vision-Language-Action (VLA) models demonstrate impressive reasoning over visual, semantic, and spatial task variations by leveraging large-scale vision and language pre-training. They remain, however, largely blind to contact forces, which seldom manifest clearly in visual feedback but are central to contact-rich manipulation. Tactile sensing measures these forces directly, but integrating it into VLAs is difficult: tactile data is absent from the large-scale corpora used to pre-train VLAs, so adding it as a new input modality induces a distribution shift that erodes the very pre-training that makes VLAs effective. We propose Tactile Annotation Prompting for Vision-Language-Action models (TAP-VLA), a simple framework that supplies tactile feedback through visual augmentation rather than architectural change. TAP-VLA extracts shear fields from visuo-tactile sensors and overlays them as spatially-grounded vectors onto the multi-view RGB images the policy already consumes, yielding a clear, interpretable tactile cue in the VLA's native observation space. Because the architecture is untouched, the approach requires no tactile pre-training, adds negligible compute, and stays close to the pre-training distribution. Across four contact-rich tasks, TAP-VLA succeeds on 78% of trials, compared to under 50% for vision-only fine-tuning and alternative tactile-fusion baselines -- including tasks where the baselines perform no better than chance.

Problem

• Vision-Language-Action models
• tactile sensing
• contact-rich manipulation
• distribution shift
• pre-training

Innovation

• Tactile Sensing
• Vision-Language-Action Models
• Visual Augmentation
• Shear Field Overlay
• Distribution Shift Mitigation