Vi-TacMan: Articulated Object Manipulation via Vision and Touch

2025-10-07

AI Summary

Autonomous manipulation of articulated objects faces dual challenges: poor visual generalization and reliance on tactile initialization. To address these, we propose a vision-guided, tactile-refined collaborative framework that operates without prior kinematic models: vision provides global pose estimation and initial grasping configurations, while tactile feedback enables local closed-loop control. We incorporate surface normal geometry as a geometric prior to constrain motion directionality and employ the von Mises–Fisher distribution to probabilistically model joint axis orientation, thereby enhancing cross-category generalization. Evaluated across over 50,000 trials in simulation and real-world settings, our method significantly outperforms baselines (p < 0.0001), demonstrating strong robustness, scalability, and zero-shot transfer capability across diverse articulated objects.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Autonomous manipulation of articulated objects remains a fundamental challenge for robots in human environments. Vision-based methods can infer hidden kinematics but can yield imprecise estimates on unfamiliar objects. Tactile approaches achieve robust control through contact feedback but require accurate initialization. This suggests a natural synergy: vision for global guidance, touch for local precision. Yet no framework systematically exploits this complementarity for generalized articulated manipulation. Here we present Vi-TacMan, which uses vision to propose grasps and coarse directions that seed a tactile controller for precise execution. By incorporating surface normals as geometric priors and modeling directions via von Mises-Fisher distributions, our approach achieves significant gains over baselines (all p<0.0001). Critically, manipulation succeeds without explicit kinematic models -- the tactile controller refines coarse visual estimates through real-time contact regulation. Tests on more than 50,000 simulated and diverse real-world objects confirm robust cross-category generalization. This work establishes that coarse visual cues suffice for reliable manipulation when coupled with tactile feedback, offering a scalable paradigm for autonomous systems in unstructured environments.

Problem

• Combining vision and touch for articulated object manipulation
• Overcoming imprecise visual estimates with tactile feedback
• Enabling model-free manipulation through real-time contact regulation

Innovation

• Vision proposes grasps and coarse directions
• Tactile controller refines estimates via contact
• Uses surface normals as geometric priors