AnyViewDex: View-Invariant Dexterous Manipulation from RGB Observations

2026-09-17

AI Summary

为解决多指灵巧操作对相机视角变化敏感的问题，本文提出AnyViewDex方法，通过在模拟训练中编码几何知识来实现仅依赖单目RGB图像的视点不变控制。

Abstract

Visuomotor policies for multi-fingered dexterous manipulation are highly sensitive to camera viewpoint shifts. To achieve view invariance, recent methods increasingly rely on explicit 3D modalities like RGB-D or point clouds, which can introduce hardware dependencies, calibration requirements, and vulnerability to sensor noise during real-world deployment. In this work, we show that view-invariant control can be achieved without explicit test-time 3D sensing by encoding geometric knowledge into the visual representation during simulation. We present AnyViewDex, an asymmetric training pipeline that combines multi-view contrastive alignment with privileged 3D geometric supervision. By regressing absolute 3D object coordinates during simulated training, this auxiliary objective provides a geometric grounding signal that mitigates the spatial collapse of the globally pooled contrastive embedding. At deployment, the policy operates zero-shot using only uncalibrated monocular RGB and proprioception. We validate this approach across both reinforcement learning and student-teacher distillation. In hardware evaluation on an xArm7 with a 16-DoF LEAP Hand, AnyViewDex reaches 76.7% grasping success across eight unseen objects and six uncalibrated viewpoints (480 trials; 2,400 across all ablation conditions), indicating that geometrically grounded monocular policies transfer zero-shot without test-time depth. Project Page: https://anyviewdex.github.io/

Problem

• view-invariant
• dexterous manipulation
• RGB observations
• multi-fingered
• visuomotor policies

Innovation

• view-invariant manipulation
• geometric grounding signal
• contrastive alignment
• privileged 3D geometric supervision
• zero-shot operation