Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policy

2024-10-02

arXiv.org

AI Summary

To address the lack of a unified benchmark for evaluating the generalization capability of language-conditioned robotic policies on novel tasks, this paper introduces GemBench—the first vision-language manipulation benchmark supporting four-level zero-shot generalization: across objects, layouts, robot configurations, and long-horizon tasks. We further propose 3D-LOTUS++, the first framework to tightly integrate large language model (LLM)-driven task reasoning, vision-language model (VLM)-based object localization, and 3D motion planning into a hierarchical task-motion coordination architecture. Experiments demonstrate that our method achieves state-of-the-art performance on GemBench, significantly improving zero-shot generalization across all four dimensions. The benchmark, source code, and models are publicly released.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Generalizing language-conditioned robotic policies to new tasks remains a significant challenge, hampered by the lack of suitable simulation benchmarks. In this paper, we address this gap by introducing GemBench, a novel benchmark to assess generalization capabilities of vision-language robotic manipulation policies. GemBench incorporates seven general action primitives and four levels of generalization, spanning novel placements, rigid and articulated objects, and complex long-horizon tasks. We evaluate state-of-the-art approaches on GemBench and also introduce a new method. Our approach 3D-LOTUS leverages rich 3D information for action prediction conditioned on language. While 3D-LOTUS excels in both efficiency and performance on seen tasks, it struggles with novel tasks. To address this, we present 3D-LOTUS++, a framework that integrates 3D-LOTUS's motion planning capabilities with the task planning capabilities of LLMs and the object grounding accuracy of VLMs. 3D-LOTUS++ achieves state-of-the-art performance on novel tasks of GemBench, setting a new standard for generalization in robotic manipulation. The benchmark, codes and trained models are available at https://www.di.ens.fr/willow/research/gembench/.

Problem

• Lack of benchmarks for vision-language robotic manipulation generalization.
• Challenges in generalizing language-conditioned policies to new tasks.
• Need for improved methods to handle novel robotic manipulation tasks.

Innovation

• Introduces GemBench for vision-language robotic manipulation.
• Develops 3D-LOTUS for 3D-based action prediction.
• Enhances 3D-LOTUS++ with LLM and VLM integration.