Pose-Agnostic Robotic Functional Grasping via Observation-Action Canonicalization

2026-06-19

AI Summary

This work addresses the challenge of robotic grasping generalization across diverse object poses and geometries—such as upright versus inverted mugs—by introducing the AnyMug framework. The approach employs observation-action normalization, aligning depth images and end-effector actions into a common object-centric coordinate frame to train a single closed-loop policy capable of precise, functional grasps on mug handles. Innovatively integrating handle-aware reward shaping, pose-based curriculum learning, and domain randomization, the method substantially enhances policy generalization and zero-shot transfer to unseen object orientations. In simulation, the policy achieves over 93% success on previously unobserved upright and inverted mugs; when transferred without fine-tuning to a real Franka Panda robot, it attains an 80% success rate across five novel mugs.

Abstract

Functional robotic grasping requires a policy that generalizes across diverse object geometries and poses while maintaining task-specific contact precision. We study this challenge through mug-handle grasping, where thin handles, instance variation, and upright or inverted placements make both perception and control sensitive to object configuration. Grasp pose detection methods operate open-loop and are sensitive to estimation errors on thin handle structures. Learned visuomotor policies must implicitly learn to handle the coupled variation in visual appearance and action direction induced by different object placements, limiting generalization. We propose AnyMug, a canonicalized visuomotor reinforcement learning framework for functional grasping that trains a single closed-loop policy entirely in simulation and deploys it zero-shot on a real robot. AnyMug introduces observation-action canonicalization, which transforms both the depth observation and the predicted end-effector action into a shared object-centric frame. The policy therefore sees a consistent mug-centered view and emits actions in a canonical direction regardless of mug placement, allowing the same grasping behavior to be reused across configurations. A handle-aware reward further encourages precise approach, gripper alignment, and opposing-finger placement, while a pose curriculum and domain randomization improve training stability and sim-to-real transfer. In simulation, AnyMug achieves over 93% success rate on both unseen upright and inverted mugs and transfers zero-shot to a real Franka Panda, reaching 80% success rate on 5 held-out physical mugs across both pose categories.

Problem

• functional grasping
• pose-agnostic
• object-centric canonicalization
• sim-to-real transfer
• visuomotor policy

Innovation

• observation-action canonicalization
• functional grasping
• sim-to-real transfer
• visuomotor policy
• object-centric frame