"Don't Do That!": Guiding Embodied Systems through Large Language Model-based Constraint Generation

2025-06-04

AI Summary

This work addresses the automatic translation of informal spatial, mathematical, and conditional constraints in natural language—e.g., “do not approach the edges”—into executable, formalized constraints for robot navigation. We propose STPR, a framework that leverages large language models (including lightweight, code-specialized LLMs) to directly generate transparent, verifiable Python constraint functions, bypassing conventional semantic parsing and significantly mitigating hallucination. STPR integrates point-cloud-based environmental modeling, graph search (A*), and efficient code generation to support complex geometric and logical constraint specification. Evaluated in Gazebo simulation, it achieves 100% compliance under multiple concurrent constraints, with low inference latency and compatibility with resource-constrained models. Our core contribution is the first end-to-end, formally verifiable paradigm for translating unstructured natural-language instructions into executable constraint code—substantially enhancing the safety and reliability of embodied agents in responding to ambiguous or informal directives.

Technology Category

• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Embodied AI
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recent advancements in large language models (LLMs) have spurred interest in robotic navigation that incorporates complex spatial, mathematical, and conditional constraints from natural language into the planning problem. Such constraints can be informal yet highly complex, making it challenging to translate into a formal description that can be passed on to a planning algorithm. In this paper, we propose STPR, a constraint generation framework that uses LLMs to translate constraints (expressed as instructions on ``what not to do'') into executable Python functions. STPR leverages the LLM's strong coding capabilities to shift the problem description from language into structured and transparent code, thus circumventing complex reasoning and avoiding potential hallucinations. We show that these LLM-generated functions accurately describe even complex mathematical constraints, and apply them to point cloud representations with traditional search algorithms. Experiments in a simulated Gazebo environment show that STPR ensures full compliance across several constraints and scenarios, while having short runtimes. We also verify that STPR can be used with smaller, code-specific LLMs, making it applicable to a wide range of compact models at low inference cost.

Problem

• Translate informal language constraints into executable code for robotic navigation
• Leverage LLMs to avoid complex reasoning and hallucinations in constraint generation
• Ensure compliance with constraints in simulated environments using generated code

Innovation

• LLM translates constraints into Python functions
• Uses code to avoid complex reasoning and hallucinations
• Applies constraints to point clouds with search algorithms