Inference-time Policy Steering via Vision and Touch

2026-06-12

AI Summary

This work addresses the low success rate of purely vision-based policies in contact-rich manipulation tasks due to the absence of tactile feedback. To overcome this limitation, the authors propose ViTaL, a framework that leverages multimodal visual-tactile verification to guide a pretrained generative policy during inference. ViTaL introduces tactile information into the inference-time guidance process for the first time, establishing a bilevel optimization mechanism: a high-level visual module selects candidate actions, while a low-level tactile diffusion editor refines them with precision. Key innovations include a text-conditioned tactile reward, a multimodal alignment verifier, and a visual-tactile latent world model. Evaluated on three real-world contact-intensive tasks, ViTaL improves overall success rates by 51% over baseline methods, substantially outperforming unimodal guidance (+33%) and naive multimodal fusion (+20%).

Abstract

Inference-time steering adapts pre-trained generative robot policies during deployment by verifying candidate actions before execution. While prior methods typically perform this verification only with visual observations, vision alone is often insufficient for contact-rich manipulation, where success depends on both global task progress and subtle local interactions such as contact force. We introduce ViTaL, a visuo-tactile inference-time steering framework that formulates multimodal guidance as a bi-level optimization problem. At the high level, visual sampling-and-verification performs long-horizon mode selection, deciding what behavior the robot should execute. At the low level, tactile-guided diffusion editing refines the selected action sequence over a shorter horizon to satisfy local contact requirements. To support outcome-based steering, ViTaL learns a visuo-tactile latent world model and employs semantically aligned visual and tactile verifiers, including a novel text-conditioned tactile reward that scores predicted tactile futures directly in latent space. Across three real-world contact-rich manipulation tasks, ViTaL improves overall success by 51% over the base policy, outperforms unimodal steering by at least 33%, and exceeds naive multimodal fusion by at least 20%. Website: https://yilin-wu98.github.io/vital_website.

Problem

• inference-time steering
• visuo-tactile perception
• contact-rich manipulation
• robot policy adaptation
• multimodal verification

Innovation

• inference-time steering
• visuo-tactile fusion
• diffusion editing
• latent world model
• tactile reward