When would Vision-Proprioception Policies Fail in Robotic Manipulation?

2026-02-12

AI Summary

This work addresses the limited generalization of visuo-proprioceptive policies in robotic manipulation, particularly during motion transitions where visual inputs struggle to contribute effectively. Through temporal control experiments, the study reveals that proprioception dominates learning due to faster optimization, thereby suppressing visual modality acquisition. To mitigate this imbalance, the authors propose Gradient Adjustment with Phase guidance (GAP), a novel algorithm that estimates the probability of motion transition phases using proprioceptive states and adaptively attenuates proprioceptive gradients during these phases to encourage collaborative visual learning. Notably, GAP requires no architectural modifications and is compatible with both conventional policies and vision-language-action models. Experiments in simulation and real-world settings demonstrate that GAP significantly enhances the robustness and generalization of both single-arm and dual-arm manipulation policies.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Multi-modal Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Proprioceptive information is critical for precise servo control by providing real-time robotic states. Its collaboration with vision is highly expected to enhance performances of the manipulation policy in complex tasks. However, recent studies have reported inconsistent observations on the generalization of vision-proprioception policies. In this work, we investigate this by conducting temporally controlled experiments. We found that during task sub-phases that robot's motion transitions, which require target localization, the vision modality of the vision-proprioception policy plays a limited role. Further analysis reveals that the policy naturally gravitates toward concise proprioceptive signals that offer faster loss reduction when training, thereby dominating the optimization and suppressing the learning of the visual modality during motion-transition phases. To alleviate this, we propose the Gradient Adjustment with Phase-guidance (GAP) algorithm that adaptively modulates the optimization of proprioception, enabling dynamic collaboration within the vision-proprioception policy. Specifically, we leverage proprioception to capture robotic states and estimate the probability of each timestep in the trajectory belonging to motion-transition phases. During policy learning, we apply fine-grained adjustment that reduces the magnitude of proprioception's gradient based on estimated probabilities, leading to robust and generalizable vision-proprioception policies. The comprehensive experiments demonstrate GAP is applicable in both simulated and real-world environments, across one-arm and dual-arm setups, and compatible with both conventional and Vision-Language-Action models. We believe this work can offer valuable insights into the development of vision-proprioception policies in robotic manipulation.

Problem

• vision-proprioception policy
• robotic manipulation
• motion-transition phases
• generalization
• target localization

Innovation

• vision-proprioception policy
• motion-transition phase
• gradient modulation
• GAP algorithm
• multimodal robot learning