Free-form language-based robotic reasoning and grasping

2025-03-17

AI Summary

This work addresses zero-shot robotic grasping guided by free-form natural language instructions in cluttered scenes. The proposed FreeGrasp framework reformulates object detection as keypoint prediction and introduces image-level spatial annotations to enhance vision-language models’ (VLMs) fine-grained understanding of semantic and spatial relationships—e.g., occlusion and support. It also introduces FreeGraspData, the first synthetic dataset featuring human-authored diverse instructions paired with corresponding grasp sequences. FreeGrasp integrates state-of-the-art VLMs (e.g., GPT-4o), keypoint-driven spatial reasoning, zero-shot cross-modal alignment, and closed-loop control on a physical robotic arm. Extensive experiments on both synthetic and real-world platforms demonstrate significant improvements in grasp feasibility assessment accuracy and execution success rate, achieving new state-of-the-art performance. Notably, FreeGrasp enables robust, language-guided grasping without task-specific fine-tuning—the first such approach to do so.

Technology Category

• Computer Vision: Language and Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Performing robotic grasping from a cluttered bin based on human instructions is a challenging task, as it requires understanding both the nuances of free-form language and the spatial relationships between objects. Vision-Language Models (VLMs) trained on web-scale data, such as GPT-4o, have demonstrated remarkable reasoning capabilities across both text and images. But can they truly be used for this task in a zero-shot setting? And what are their limitations? In this paper, we explore these research questions via the free-form language-based robotic grasping task, and propose a novel method, FreeGrasp, leveraging the pre-trained VLMs' world knowledge to reason about human instructions and object spatial arrangements. Our method detects all objects as keypoints and uses these keypoints to annotate marks on images, aiming to facilitate GPT-4o's zero-shot spatial reasoning. This allows our method to determine whether a requested object is directly graspable or if other objects must be grasped and removed first. Since no existing dataset is specifically designed for this task, we introduce a synthetic dataset FreeGraspData by extending the MetaGraspNetV2 dataset with human-annotated instructions and ground-truth grasping sequences. We conduct extensive analyses with both FreeGraspData and real-world validation with a gripper-equipped robotic arm, demonstrating state-of-the-art performance in grasp reasoning and execution. Project website: https://tev-fbk.github.io/FreeGrasp/.

Problem

• Robotic grasping from cluttered bins using human instructions
• Zero-shot application of Vision-Language Models for spatial reasoning
• Development of a synthetic dataset for free-form language-based grasping

Innovation

• Leverages pre-trained Vision-Language Models for reasoning
• Uses keypoints to annotate images for spatial reasoning
• Introduces synthetic dataset FreeGraspData for validation