GraspIT: A Dataset Bridging the Sim-to-Real gap and back for Validated Grasping SE(3) Pose Generation

2026-07-07

AI Summary

This work addresses the limitations of existing grasp datasets, which lack physically validated continuous quality annotations and simulation-to-reality alignment, thereby hindering robust grasping on novel objects. The authors construct tabletop scenes in NVIDIA Isaac Sim and generate grasps for the Franka Panda manipulator, evaluating each with trajectory feasibility and a continuous quality score derived from a four-stage physical slip test and force-closure analysis. A Real↔Sim closed-loop framework enables bidirectional domain alignment. The resulting dataset is the first to jointly provide physically validated continuous grasp quality labels, hard negative samples with difficulty tiers, and high-fidelity 6-DoF pose annotations. It comprises 316,000 RGB-D frames spanning 1,035 simulated and 100 real-world scenes, along with 2.3 million candidate grasps—83% high-quality and 17% hard negatives—and includes a fully open-sourced toolchain.

Abstract

Robust robotic grasping of novel objects requires datasets that simultaneously provide photorealistic RGB-D observations, physically validated grasp quality annotations, and a principled bridge between simulation and the real world, which existing datasets lack to provide jointly. \textbf{GraspIT} addresses this gap: tabletop scenes in NVIDIA Isaac Sim are annotated via a four-stage physical slip-test on parallel Franka Panda instances, producing trajectory-reachability checks and continuous quality scores beyond force-closure.Of ${\sim}$2.3M candidates, 83% pass as \emph{good} ($s{\geq}0.50$); the 17% that passed force-closure but failed the slip-test provide graded hard negatives. A Real$\leftrightarrow$Sim loop back-projects these labels onto 100 real-world scenes. The release provides ${\sim}$316k annotated RGBD frame sets across 1035 sim and 100 real scenes, with instance masks, 6-DoF poses, physical object properties, and scored 6-DoF grasps. All tools are open-source and Docker-containerized. The trajectory planning within Isaac Sim further allows streaming of high resolution demonstrations for tabletop manipulation policy learning and behavior cloning.

Problem

• grasping
• sim-to-real
• RGB-D
• physical validation
• SE(3) pose

Innovation

• sim-to-real
• grasp validation
• slip-test
• 6-DoF grasp
• physical simulation