VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAs

2026-03-24

AI Summary

Existing video-based action models struggle to accurately model force control and contact states in contact-intensive tasks due to the absence of tactile information, often resulting in unstable behaviors. To address this limitation, this work proposes a multimodal world model that integrates visual and tactile modalities by injecting tactile signals into a pretrained video Transformer through lightweight modality-transfer fine-tuning. A tactile regularization loss is further introduced to balance cross-modal attention and mitigate visual dominance. Notably, this approach achieves efficient joint video-tactile representation learning without requiring paired tactile-language data or separate tactile pretraining. Evaluated on contact-intensive manipulation tasks, the method attains an average success rate of 90% and demonstrates an 80% improvement over the Pi 0.5 baseline on high-precision force-control tasks such as chip grasping and placement.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Video-Action Models (VAMs) have emerged as a promising framework for embodied intelligence, learning implicit world dynamics from raw video streams to produce temporally consistent action predictions. Although such models demonstrate strong performance on long-horizon tasks through visual reasoning, they remain limited in contact-rich scenarios where critical interaction states are only partially observable from vision alone. In particular, fine-grained force modulation and contact transitions are not reliably encoded in visual tokens, leading to unstable or imprecise behaviors. To bridge this gap, we introduce the Video-Tactile Action Model (VTAM), a multimodal world modeling framework that incorporates tactile perception as a complementary grounding signal. VTAM augments a pretrained video transformer with tactile streams via a lightweight modality transfer finetuning, enabling efficient cross-modal representation learning without tactile-language paired data or independent tactile pretraining. To stabilize multimodal fusion, we introduce a tactile regularization loss that enforces balanced cross-modal attention, preventing visual latent dominance in the action model. VTAM demonstrates superior performance in contact-rich manipulation, maintaining a robust success rate of 90 percent on average. In challenging scenarios such as potato chip pick-and-place requiring high-fidelity force awareness, VTAM outperforms the pi 0.5 baseline by 80 percent. Our findings demonstrate that integrating tactile feedback is essential for correcting visual estimation errors in world action models, providing a scalable approach to physically grounded embodied foundation models.

Problem

• Video-Action Models
• tactile perception
• contact-rich manipulation
• multimodal fusion
• force modulation

Innovation

• Video-Tactile-Action Model
• multimodal fusion
• tactile regularization
• embodied intelligence
• modality transfer fine-tuning