Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answering

2025-07-17

AI Summary

This paper addresses Long-horizon Active Embodied Question Answering (LA-EQA), where an embodied agent must reason across temporal dimensions—past, present, and future—to answer temporally complex questions via active exploration and memory-augmented reasoning. To overcome limitations of existing EQA methods—including insufficient long-range memory modeling and decoupled exploration-decision mechanisms—we propose: (1) a structured scene graph memory system inspired by the “method of loci,” enabling goal-directed episodic memory retrieval and semantic navigation; and (2) an information-gain-based adaptive stopping mechanism that dynamically balances exploration cost against answer confidence. Evaluated in both simulation and real-world environments, our approach achieves significant improvements: +18.7% absolute gain in answer accuracy and a 32.4% reduction in path length, demonstrating enhanced exploration efficiency. To our knowledge, this is the first work to realize closed-loop optimization of perception–memory–decision across temporal horizons for LA-EQA.

Technology Category

• Natural Language Processing: Question Answering
• Search and Optimization: Learning to Search
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

As robots become increasingly capable of operating over extended periods -- spanning days, weeks, and even months -- they are expected to accumulate knowledge of their environments and leverage this experience to assist humans more effectively. This paper studies the problem of Long-term Active Embodied Question Answering (LA-EQA), a new task in which a robot must both recall past experiences and actively explore its environment to answer complex, temporally-grounded questions. Unlike traditional EQA settings, which typically focus either on understanding the present environment alone or on recalling a single past observation, LA-EQA challenges an agent to reason over past, present, and possible future states, deciding when to explore, when to consult its memory, and when to stop gathering observations and provide a final answer. Standard EQA approaches based on large models struggle in this setting due to limited context windows, absence of persistent memory, and an inability to combine memory recall with active exploration. To address this, we propose a structured memory system for robots, inspired by the mind palace method from cognitive science. Our method encodes episodic experiences as scene-graph-based world instances, forming a reasoning and planning algorithm that enables targeted memory retrieval and guided navigation. To balance the exploration-recall trade-off, we introduce value-of-information-based stopping criteria that determines when the agent has gathered sufficient information. We evaluate our method on real-world experiments and introduce a new benchmark that spans popular simulation environments and actual industrial sites. Our approach significantly outperforms state-of-the-art baselines, yielding substantial gains in both answer accuracy and exploration efficiency.

Problem

• Robots need long-term memory and exploration for complex questions
• Current EQA methods fail with memory and exploration integration
• Propose structured memory system for efficient reasoning and planning

Innovation

• Structured memory system using scene-graph-based encoding
• Value-of-information-based stopping criteria for exploration
• Combined reasoning and planning algorithm for navigation