SG-CoT: An Ambiguity-Aware Robotic Planning Framework using Scene Graph Representations

2026-03-18

AI Summary

This work addresses the challenge of planning failures in robotic systems caused by ambiguous natural language instructions when leveraging large language models (LLMs). To mitigate this issue, the authors propose SG-CoT, a novel framework that uniquely integrates scene graphs with chain-of-thought (CoT) reasoning. SG-CoT constructs a structured scene graph representation of the environment and guides the LLM to iteratively retrieve relevant information from the graph, enabling it to detect ambiguities and proactively generate clarification questions. This approach synergistically combines scene graph construction, LLM-based retrieval, CoT reasoning, and multi-agent interaction mechanisms. Empirical results demonstrate that SG-CoT improves task success rates by at least 4% in single-agent settings and by 15% in multi-agent environments, while also achieving over a 10% increase in ambiguity identification accuracy.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Ambiguity poses a major challenge to large language models (LLMs) used as robotic planners. In this letter, we present Scene Graph-Chain-of-Thought (SG-CoT), a two-stage framework where LLMs iteratively query a scene graph representation of the environment to detect and clarify ambiguities. First, a structured scene graph representation of the environment is constructed from input observations, capturing objects, their attributes, and relationships with other objects. Second, the LLM is equipped with retrieval functions to query portions of the scene graph that are relevant to the provided instruction. This grounds the reasoning process of the LLM in the observation, increasing the reliability of robotic planners under ambiguous situations. SG-CoT also allows the LLM to identify the source of ambiguity and pose a relevant disambiguation question to the user or another robot. Extensive experimentation demonstrates that SG-CoT consistently outperforms prior methods, with a minimum of 10% improvement in question accuracy and a minimum success rate increase of 4% in single-agent and 15% in multi-agent environments, validating its effectiveness for more generalizable robot planning.

Problem

• ambiguity
• robotic planning
• large language models
• scene understanding
• disambiguation

Innovation

• Scene Graph
• Chain-of-Thought
• Ambiguity Resolution
• Robotic Planning
• Large Language Models