In-Context Learning Enables Robot Action Prediction in LLMs

2024-10-16

arXiv.org

AI Summary

Bridging the modality gap between vision-action data in embodied tasks and purely text-based large language models (LLMs) remains challenging for zero-training robotic action prediction. Method: We propose RoboPrompt, a framework that enables direct LLM-driven action generation without fine-tuning or visual encoders. It employs keyframe heuristics for scene selection and lossless textual encoding of end-effector actions and object poses, constructing structured in-context learning (ICL) templates that map multimodal embodied information to LLM-compatible textual prompts end-to-end. Contribution/Results: Evaluated on both simulation and real-robot tasks, RoboPrompt achieves up to 37% higher action prediction accuracy than zero-shot and conventional ICL baselines. It demonstrates, for the first time, the feasibility of using off-the-shelf, purely textual LLMs—such as Llama-3 and GPT-3.5—for embodied action reasoning without any training, parameter updates, or auxiliary vision modules.

Technology Category

• Intelligent Robots: Embodied AI
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recently, Large Language Models (LLMs) have achieved remarkable success using in-context learning (ICL) in the language domain. However, leveraging the ICL capabilities within LLMs to directly predict robot actions remains largely unexplored. In this paper, we introduce RoboPrompt, a framework that enables off-the-shelf text-only LLMs to directly predict robot actions through ICL without training. Our approach first heuristically identifies keyframes that capture important moments from an episode. Next, we extract end-effector actions from these keyframes as well as the estimated initial object poses, and both are converted into textual descriptions. Finally, we construct a structured template to form ICL demonstrations from these textual descriptions and a task instruction. This enables an LLM to directly predict robot actions at test time. Through extensive experiments and analysis, RoboPrompt shows stronger performance over zero-shot and ICL baselines in simulated and real-world settings.

Problem

• Predict robot actions using LLMs without training
• Leverage in-context learning for robot action prediction
• Convert keyframes and poses into textual descriptions for LLMs

Innovation

• Uses in-context learning for robot action prediction
• Converts keyframes and poses into textual descriptions
• Enables LLMs to predict actions without training