Object Pose and Shape Estimation for Grasping: Does it Work?

2026-05-26

AI Summary

This work systematically evaluates the efficacy of modular grasping pipelines in real-world scenarios, investigating whether object pose and shape estimation–based approaches outperform end-to-end grasp synthesis. Focusing on parallel-jaw and 7-DoF grasping from single-view RGB(-D) inputs, we integrate state-of-the-art models such as SAM3D and InstantMesh for pose and shape reconstruction, complemented by antipodal grasp sampling. Furthermore, we incorporate vision-language models like LERF to enable language-conditioned grasp generation from a single viewpoint. Experimental results demonstrate that the proposed modular approach significantly surpasses end-to-end baselines in both overall and small-object grasping performance, while achieving language-guided grasping accuracy on par with LERF-TOGO. This study provides the first empirical validation of the superiority and practicality of the modular paradigm in complex robotic grasping tasks.

Abstract

The problem of object pose and shape estimation has seen key advancements lately. Encoder-decoder (e.g., SAM3D, LRM, CRISP) and diffusion-based models (e.g., InstantMesh, Zero123, SceneComplete) have shown category-agnostic shape encoding capacity and open-set generalizability. In this work, we ask the question: Are the object pose and shape estimation methods mature enough, such that when used with antipodal grasp sampling, can outperform the end-to-end grasp synthesis methods? We explore this question in detail by scoping our study to parallel jaw grippers, 7-DoF grasps, and single-view RGB(-D) image as input. We implement and compare a state-of-the-art, end-to-end grasp synthesis method and three modular methods, which first estimate the object pose and shape for all objects in the scene, and generate grasps using antipodal sampling. We observe that the modular methods outperform the end-to-end method in all our experiments. The modular methods are able to synthesize plenty of grasps, even for small objects, where the end-to-end methods fail. The effectiveness of the modular methods is contingent on the accuracy of the pose and shape estimation, and suffers partial degradation in cluttered scenes - a limitation of the existing pose and shape estimation methods. We also analyze the failure modes and run-times for the three modular methods, which use two different ways of object pose and shape estimation: one based on an encoder-decoder model, while another a diffusion model. Finally, we demonstrate that the single-view object pose and shape estimation methods can be augmented with vision-language models to yield language-conditioned grasps from just single-view RGB-D image as input. We notice comparable performance to the state-of-the-art LERF-TOGO baseline.

Problem

• object pose estimation
• shape estimation
• grasp synthesis
• modular methods
• end-to-end learning

Innovation

• modular grasp synthesis
• object pose and shape estimation
• antipodal grasp sampling
• vision-language models
• single-view RGB-D grasping