ConceptBot: Enhancing Robot's Autonomy through Task Decomposition with Large Language Models and Knowledge Graph

2025-08-30

AI Summary

To address insufficient robot autonomy stemming from ambiguous natural-language instructions and inaccurate environmental perception, this paper proposes an LLM-KG collaborative framework. It employs a large language model (LLM) as the core task planner, augmented with ConceptNet knowledge graph to enhance commonsense-aware scene understanding. The framework incorporates three key modules: object attribute extraction, instruction disambiguation, and risk-aware planning—enabling implicit intent inference and safety-sensitive decision-making. Its novelty lies in explicitly injecting structured commonsense knowledge into the LLM’s reasoning process, thereby supporting context-aware task decomposition and dynamic risk assessment. Experiments on the SafeAgentBench benchmark demonstrate an overall score of 80%, substantially outperforming prior methods. Explicit and implicit task success rates reach 100% and 87%, respectively, while risk-aware task accuracy attains 76%.

Technology Category

• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

ConceptBot is a modular robotic planning framework that combines Large Language Models and Knowledge Graphs to generate feasible and risk-aware plans despite ambiguities in natural language instructions and correctly analyzing the objects present in the environment - challenges that typically arise from a lack of commonsense reasoning. To do that, ConceptBot integrates (i) an Object Property Extraction (OPE) module that enriches scene understanding with semantic concepts from ConceptNet, (ii) a User Request Processing (URP) module that disambiguates and structures instructions, and (iii) a Planner that generates context-aware, feasible pick-and-place policies. In comparative evaluations against Google SayCan, ConceptBot achieved 100% success on explicit tasks, maintained 87% accuracy on implicit tasks (versus 31% for SayCan), reached 76% on risk-aware tasks (versus 15%), and outperformed SayCan in application-specific scenarios, including material classification (70% vs. 20%) and toxicity detection (86% vs. 36%). On SafeAgentBench, ConceptBot achieved an overall score of 80% (versus 46% for the next-best baseline). These results, validated in both simulation and laboratory experiments, demonstrate ConceptBot's ability to generalize without domain-specific training and to significantly improve the reliability of robotic policies in unstructured environments. Website: https://sites.google.com/view/conceptbot

Problem

• Resolving ambiguities in natural language robot instructions
• Enhancing scene understanding and object property analysis
• Generating risk-aware and feasible robotic task plans

Innovation

• Combines LLMs and Knowledge Graphs for planning
• Extracts object properties using ConceptNet semantics
• Generates context-aware pick-and-place policies