VT-WAM: Visual-Tactile World Action Model for Contact-Rich Manipulation

2026-07-02

AI Summary

This work addresses the challenge of leveraging sparse and visually imperceptible tactile signals for real-time response in contact-rich manipulation tasks. The authors propose VT-WAM, a flow-matching-based multimodal world model that jointly predicts future visual observations, tactile deformations, and actions. Key innovations include an asymmetric hybrid Transformer attention mechanism that aligns initial visual anchors with temporal tactile dynamics, and a contact-gated Action-Visual-Tactile Attention Guidance (AVTAG) module that adaptively increases reliance on tactile evidence during contact phases. Evaluated on six real-world tasks, VT-WAM achieves an average success rate of 71.67%, outperforming Fast-WAM and OmniVTLA by 26.67% and 35.84%, respectively.

Abstract

Contact-rich manipulation requires policies to react to local deformation, pressure, slip, and friction, yet these cues are temporally sparse and often invisible in visual observations. Existing visual-tactile policies usually feed tactile observations directly into action prediction, but rarely model tactile deformation dynamics during action generation. In this paper, we introduce VT-WAM, a Visual-Tactile World Action Model that jointly learns future visual prediction, tactile deformation prediction, and action prediction within a unified flow matching framework. In particular, VT-WAM introduces (1) Asymmetric Mixture-of-Transformers (MoT) attention to bridge a first-frame visual anchor with temporal tactile dynamics, and (2) contact-gated Action-Visual-Tactile Attention Guidance (AVTAG) to encourage action queries to rely on tactile evidence during contact phases. Across six real-world contact-rich manipulation tasks, VT-WAM achieves a 71.67% average success rate, outperforming Fast-WAM by 26.67% and OmniVTLA by 35.84%. Ablations demonstrate that modeling tactile deformation dynamics and guiding contact-phase tactile attention are both important for contact-rich tasks. Project website: https://vt-wam.github.io/.

Problem

• contact-rich manipulation
• tactile deformation
• visual-tactile policies
• action prediction
• tactile dynamics

Innovation

• Visual-Tactile World Model
• Tactile Deformation Dynamics
• Asymmetric Mixture-of-Transformers
• Contact-Gated Attention
• Flow Matching