Verifiably Following Complex Robot Instructions with Foundation Models

2024-02-18

arXiv.org

AI Summary

This work addresses the challenge of enabling robots to reliably execute open-ended, complex natural-language instructions in real-world environments. We propose LIMP, a verifiable execution framework that operates without pre-built semantic maps and supports flexible constraints and arbitrary landmark references. Its core innovation is the first symbolic instruction representation that explicitly models alignment between user intent and robot behavior, integrating foundation-model-based semantic understanding, symbolic logical grounding, motion planning, and formal verification into an end-to-end interpretable and verifiable pipeline. Evaluated on 150 spatiotemporal instructions across five real-world scenarios, LIMP achieves a 79% success rate—significantly surpassing the strongest baseline (38%)—and establishes new state-of-the-art performance on open-vocabulary instruction-following tasks.

Technology Category

• Intelligent Robots: Motion and Path Planning
• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

When instructing robots, users want to flexibly express constraints, refer to arbitrary landmarks, and verify robot behavior, while robots must disambiguate instructions into specifications and ground instruction referents in the real world. To address this problem, we propose Language Instruction grounding for Motion Planning (LIMP), an approach that enables robots to verifiably follow complex, open-ended instructions in real-world environments without prebuilt semantic maps. LIMP constructs a symbolic instruction representation that reveals the robot's alignment with an instructor's intended motives and affords the synthesis of correct-by-construction robot behaviors. We conduct a large-scale evaluation of LIMP on 150 instructions across five real-world environments, demonstrating its versatility and ease of deployment in diverse, unstructured domains. LIMP performs comparably to state-of-the-art baselines on standard open-vocabulary tasks and additionally achieves a 79% success rate on complex spatiotemporal instructions, significantly outperforming baselines that only reach 38%. See supplementary materials and demo videos at https://robotlimp.github.io

Problem

• Robots must disambiguate and ground complex user instructions.
• Need flexible constraint expression and behavior verification for robots.
• Requires real-world instruction following without prebuilt semantic maps.

Innovation

• Uses symbolic instruction representation for alignment
• Synthesizes correct-by-construction robot behaviors
• Operates without prebuilt semantic maps