Tactile and Vision Conditioned Contact-Centric Control for Whole-Arm Manipulation

2026-07-10

AI Summary

This work addresses key challenges in whole-arm manipulation, including partial observability of multi-link contact states, data sparsity, and physically inconsistent policies under distribution shift. To this end, the authors propose TACTIC, a receding-horizon controller that fuses RGB-D vision, distributed tactile sensing, and 2D proximity perception. TACTIC introduces a contact-centric hybrid prediction architecture, combining a learned implicit dynamics model with analytically derived kinematics based on contact Jacobians, and incorporates a contact-aware action sampling mechanism. Within a model predictive control (MPC) framework, it jointly optimizes task progress and whole-arm interaction forces. Experiments demonstrate that TACTIC significantly outperforms existing model-based and model-free baselines in simulation and successfully executes complex multi-contact tasks on a real robot, including flipping a human mannequin, repositioning objects, and navigating a 3D dynamic maze to reach target locations.

Abstract

Whole-arm manipulation involves direct contact with the environment while the robot completes a task by distributing contact across multiple links as contacts form, slide, and break. This setting breaks common implicit assumptions in many learning-based manipulation pipelines: arm configuration tightly couples motion and contact forces, contact state is partially observed under occlusion, and purely learned rollouts can become physically inconsistent under distribution shift because many multi-link contact configurations are sparsely represented in the data. To address this, we propose TACTIC (Tactile and Vision Conditioned Contact-Centric Control), a receding-horizon controller for whole-arm manipulation. TACTIC uses a contact-centric hybrid predictive model that combines RGB-D, distributed tactile sensing, and a compact 2D proximity representation. The model couples a learned, action-conditioned latent dynamics model with analytical kinematics through contact Jacobians, enabling rollouts of future contact configurations and interaction forces. TACTIC integrates these rollouts into a sampling-based MPC planner with contact-aware action sampling: contact Jacobian-based projections steer sampled action sequences toward force-modulating directions, and objectives defined over predicted proximity and interaction forces trade task progress against whole-arm force regulation. We evaluate TACTIC in simulation against state-of-the-art model-based and model-free methods, and perform ablations that isolate the contribution of each design choice. TACTIC consistently outperforms other methods. We further demonstrate real-world performance on a robot with distributed tactile sensing across three whole-arm manipulation tasks that require multi-contact trajectories: turning over and repositioning a manikin, and goal-reaching in a 3D dynamic maze. Website: https://emprise.cs.cornell.edu/tactic

Problem

• whole-arm manipulation
• multi-contact interaction
• contact state estimation
• distribution shift
• physical consistency

Innovation

• whole-arm manipulation
• contact-centric control
• tactile sensing
• hybrid predictive model
• contact Jacobian