Ontology-driven Prompt Tuning for LLM-based Task and Motion Planning

2024-12-10

arXiv.org

AI Summary

Existing LLM-driven Task and Motion Planning (TAMP) in dynamic environments suffers from semantic drift, temporal logic violations, and poor environmental adaptability due to static, template-based prompting. Method: This paper proposes an ontology-driven prompt tuning framework that integrates domain ontology modeling and knowledge graph reasoning into the LLM prompting process, enabling environment state awareness, task-contextual reasoning, and symbolic knowledge injection—thereby overcoming the limitations of rigid prompt templates. Contribution/Results: We introduce the first dynamic, semantically interpretable, ontology-guided prompting mechanism, tightly coupled with the TAMP architecture. Experiments on both simulation and real-robot platforms demonstrate significant improvements in planning correctness and dynamic adaptability; notably, for complex hierarchical object placement tasks, the method ensures temporal consistency and semantic validity.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Performing complex manipulation tasks in dynamic environments requires efficient Task and Motion Planning (TAMP) approaches, which combine high-level symbolic plan with low-level motion planning. Advances in Large Language Models (LLMs), such as GPT-4, are transforming task planning by offering natural language as an intuitive and flexible way to describe tasks, generate symbolic plans, and reason. However, the effectiveness of LLM-based TAMP approaches is limited due to static and template-based prompting, which struggles in adapting to dynamic environments and complex task contexts. To address these limitations, this work proposes a novel ontology-driven prompt-tuning framework that employs knowledge-based reasoning to refine and expand user prompts with task contextual reasoning and knowledge-based environment state descriptions. Integrating domain-specific knowledge into the prompt ensures semantically accurate and context-aware task plans. The proposed framework demonstrates its effectiveness by resolving semantic errors in symbolic plan generation, such as maintaining logical temporal goal ordering in scenarios involving hierarchical object placement. The proposed framework is validated through both simulation and real-world scenarios, demonstrating significant improvements over the baseline approach in terms of adaptability to dynamic environments, and the generation of semantically correct task plans.

Problem

• Enhancing TAMP adaptability in dynamic environments using LLMs
• Overcoming static prompting limitations in LLM-based task planning
• Ensuring semantically accurate plans via knowledge-based reasoning

Innovation

• Knowledge-based reasoning refines user prompts
• Integrates domain-specific knowledge for accuracy
• Combines LLMs with dynamic environment adaptability