TouchWorld: A Predictive and Reactive Tactile Foundation Model for Dexterous Manipulation

2026-07-08

AI Summary

Dexterous manipulation requires seamless integration of high-level task planning and low-level contact responses, yet existing approaches struggle to effectively fuse tactile feedback with semantic instructions and exhibit limited robustness under perturbations. This work proposes a hierarchical policy architecture that, for the first time, leverages tactile signals simultaneously for predictive contact modeling and high-frequency residual correction, thereby decoupling slow visual-language subtask planning from rapid tactile reactions. The framework integrates vision, language, touch, and proprioception, achieving a success rate of 65.0% in clean conditions and 53.7% under human-induced disturbances across six long-horizon tasks with high contact complexity—outperforming the strongest baseline by 15.7 and 18.5 percentage points, respectively.

Abstract

Dexterous manipulation in everyday environments requires both anticipation and reaction: a robot must predict how contact should evolve while rapidly correcting local errors caused by slip, misalignment, unstable grasping, or force mismatch. Vision and language provide semantic and geometric guidance, but they cannot reliably reveal hidden contact states such as force, slip, and contact stability. Although tactile sensing exposes these physical cues, most existing policies treat touch as a low-frequency observation stream within a monolithic action model, coupling slow task reasoning, action generation, and fast contact feedback in a single loop. We introduce TouchWorld, a predictive-and-reactive tactile foundation model for dexterous manipulation. TouchWorld uses a hierarchical policy that separates vision-language subtask planning, tactile world-model prediction, visuo-tactile goal-conditioned action generation, and high-frequency tactile residual refinement. A High-Level Planning Layer produces executable subtasks and predicts tactile subgoals; a Visuo-Tactile Goal-Conditioned Policy generates nominal action chunks; and a Tactile-Conditioned Refinement Policy performs online residual correction using recent tactile and proprioceptive feedback. By using touch as both a predictive contact reference and a fast feedback signal, TouchWorld preserves the semantic generalization of vision-language-action policies while improving local contact adaptation. Across six long-horizon and contact-rich dexterous manipulation tasks, TouchWorld achieves 65.0% success in the clean setting and 53.7% success under human perturbations, outperforming the strongest baseline by 15.7 and 18.5 percentage points, respectively.

Problem

• dexterous manipulation
• tactile sensing
• contact stability
• slip detection
• force feedback

Innovation

• tactile foundation model
• hierarchical policy
• predictive-and-reactive control
• dexterous manipulation
• visuo-tactile integration