AdaptBot: Combining LLM with Knowledge Graphs and Human Input for Generic-to-Specific Task Decomposition and Knowledge Refinement

2025-02-04

AI Summary

To address the challenge of rapid adaptation for embodied agents in novel tasks and environments—where labeled data and training resources are scarce—this paper proposes a tripartite synergistic framework integrating large language models (LLMs), knowledge graphs (KGs), and dynamic human-in-the-loop interaction. The framework leverages structured KGs to constrain and ground LLM-generated actions, while incorporating real-time human feedback to enable closed-loop reasoning and continual knowledge refinement. This design bridges the semantic gap between high-level natural language instructions and executable low-level actions, and mitigates domain-infeasibility issues inherent in purely LLM-based approaches. Evaluated on simulated kitchen and cleaning tasks, our method achieves significantly higher task completion rates compared to LLM-only baselines. Results demonstrate that synergistic modeling of heterogeneous knowledge sources—symbolic (KG), statistical (LLM), and interactive (human feedback)—critically enhances the robustness and generalizability of embodied reasoning.

Technology Category

• Intelligent Robots: Embodied AI
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Embodied agents assisting humans are often asked to complete a new task in a new scenario. An agent preparing a particular dish in the kitchen based on a known recipe may be asked to prepare a new dish or to perform cleaning tasks in the storeroom. There may not be sufficient resources, e.g., time or labeled examples, to train the agent for these new situations. Large Language Models (LLMs) trained on considerable knowledge across many domains are able to predict a sequence of abstract actions for such new tasks and scenarios, although it may not be possible for the agent to execute this action sequence due to task-, agent-, or domain-specific constraints. Our framework addresses these challenges by leveraging the generic predictions provided by LLM and the prior domain-specific knowledge encoded in a Knowledge Graph (KG), enabling an agent to quickly adapt to new tasks and scenarios. The robot also solicits and uses human input as needed to refine its existing knowledge. Based on experimental evaluation over cooking and cleaning tasks in simulation domains, we demonstrate that the interplay between LLM, KG, and human input leads to substantial performance gains compared with just using the LLM output.

Problem

• Adapting agents to new tasks in new scenarios
• Combining LLM with Knowledge Graphs for task decomposition
• Using human input to refine agent knowledge

Innovation

• Combines LLM with Knowledge Graphs
• Utilizes human input for refinement
• Enables quick task adaptation