Rotation-Aware Point-Cloud Embeddings for Vision-Based In-Hand Reorientation

2026-06-19

AI Summary

This work addresses the challenge of unstable orientation cues in dexterous hand reorientation tasks due to the unordered nature, inconsistent sampling, and occlusion inherent in point cloud observations. To overcome this, the authors propose a rotation-aware point cloud embedding whose Euclidean distances in the latent space align with the SO(3) geodesic error of object orientations, thereby providing a smooth and geometrically consistent control signal for reinforcement learning policies. Notably, this approach is the first to intrinsically embed rotational geometry at the representation level, eliminating the need for external modules such as pose estimators, optical flow, or teacher supervision. Experiments demonstrate that the method achieves performance on par with baselines relying on privileged state information or distillation, while avoiding their fragile dependence on structured pose estimates or dense optical flow during testing.

Abstract

Point-cloud goals provide a direct way to specify dexterous in-hand reorientation: instead of defining an object-specific pose frame or estimating 6D pose at test time, the policy is given the desired 3D geometry of the object. Yet raw point-cloud goal conditioning is poorly conditioned for policy learning. Current and goal clouds are unordered, independently sampled, and often visibility-dependent, so their discrepancy entangles object rotation with permutation, resampling, and unstable correspondence structure. For this reason, prior point-cloud manipulation methods typically add structure outside the representation itself, such as explicit pose or relative-pose inputs, dense flow features, or distillation from privileged teachers. We close this gap by learning a rotation-aware point-cloud embedding whose Euclidean latent distance is calibrated to the SO(3) geodesic error between object orientations. The resulting representation turns current-goal comparison into a smooth control signal, allowing a model-free RL policy to act from current and goal point-cloud embeddings, proprioception, and centroid metadata, without object pose, relative pose, dense flow, or teacher-action supervision. In in-hand reorientation experiments, this interface matches privileged-state and distillation-based baselines while avoiding brittle test-time computation of structured pose or flow inputs. These results suggest that point-cloud goals become practical for this task when the representation, rather than an external module, encodes the task-relevant geometry of rotation. We also show evidence that generic visual point-cloud pretraining is insufficient for such a current-goal comparison because it discards the task-relevant state and preserves only shape features.

Problem

• point-cloud goals
• in-hand reorientation
• rotation representation
• policy learning
• SO(3) geodesic error

Innovation

• rotation-aware embedding
• point-cloud goals
• SO(3) geodesic distance
• in-hand reorientation
• model-free RL