Visual Environment-Interactive Planning for Embodied Complex-Question Answering

2025-04-01

IEEE transactions on circuits and systems for video technology (Print)

AI Summary

Embodied robots face significant challenges in answering structured, abstract natural-language questions within complex visual environments. Method: We propose an environment-driven, multi-step interactive planning framework. It constructs a hierarchical visual scene graph to parse question semantics, introduces a chain-based essential question representation for intent modeling, and integrates external rules with real-time visual feedback for closed-loop sequential decision-making. Crucially, we pioneer a structured semantic space that enables iterative vision–language interaction, eliminating reliance on single-step large language models. Contribution/Results: Evaluated on a newly constructed complex embodied question-answering dataset, our approach substantially improves planning interpretability, adaptability, and robustness. Empirical results demonstrate its feasibility and practicality in realistic embodied settings.

Technology Category

• Intelligent Robots: Embodied AI
• Natural Language Processing: Question Answering
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

This study focuses on Embodied Complex-Question Answering task, which means the embodied robot need to understand human questions with intricate structures and abstract semantics. The core of this task lies in making appropriate plans based on the perception of the visual environment. Existing methods often generate plans in a once-for-all manner, i.e., one-step planning. Such approach rely on large models, without sufficient understanding of the environment. Considering multi-step planning, the framework for formulating plans in a sequential manner is proposed in this paper. To ensure the ability of our framework to tackle complex questions, we create a structured semantic space, where hierarchical visual perception and chain expression of the question essence can achieve iterative interaction. This space makes sequential task planning possible. Within the framework, we first parse human natural language based on a visual hierarchical scene graph, which can clarify the intention of the question. Then, we incorporate external rules to make a plan for current step, weakening the reliance on large models. Every plan is generated based on feedback from visual perception, with multiple rounds of interaction until an answer is obtained. This approach enables continuous feedback and adjustment, allowing the robot to optimize its action strategy. To test our framework, we contribute a new dataset with more complex questions. Experimental results demonstrate that our approach performs excellently and stably on complex tasks. And also, the feasibility of our approach in real-world scenarios has been established, indicating its practical applicability.

Problem

• Planning for embodied robots answering complex questions
• Multi-step interactive planning with visual feedback
• Reducing reliance on large models via structured semantics

Innovation

• Sequential multi-step planning framework
• Structured semantic space for interaction
• Visual hierarchical scene graph parsing