Meta-Memory: Retrieving and Integrating Semantic-Spatial Memories for Robot Spatial Reasoning

2025-09-25

AI Summary

Robots still face challenges in efficiently constructing and retrieving semantic-spatial joint memory for natural language–based location queries in complex environments—particularly due to fragmented memory representations and the absence of effective retrieval mechanisms. This paper introduces Meta-Memory, the first large language model–based memory agent framework enabling end-to-end co-reasoning over semantics and spatial structure. Its core innovations include: (i) a generalizable semantic-spatial joint embedding space; (ii) a modality-aligned memory retrieval mechanism; and (iii) incremental organization of high-density environmental memory with natural language–driven, precise question answering. Meta-Memory achieves state-of-the-art performance on SpaceLocQA and NaVQA benchmarks and is successfully deployed on a real mobile robot platform, demonstrating robustness and practicality in open-world settings.

Technology Category

• Intelligent Robots: Localization, Mapping, and Navigation
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Navigating complex environments requires robots to effectively store observations as memories and leverage them to answer human queries about spatial locations, which is a critical yet underexplored research challenge. While prior work has made progress in constructing robotic memory, few have addressed the principled mechanisms needed for efficient memory retrieval and integration. To bridge this gap, we propose Meta-Memory, a large language model (LLM)-driven agent that constructs a high-density memory representation of the environment. The key innovation of Meta-Memory lies in its capacity to retrieve and integrate relevant memories through joint reasoning over semantic and spatial modalities in response to natural language location queries, thereby empowering robots with robust and accurate spatial reasoning capabilities. To evaluate its performance, we introduce SpaceLocQA, a large-scale dataset encompassing diverse real-world spatial question-answering scenarios. Experimental results show that Meta-Memory significantly outperforms state-of-the-art methods on both the SpaceLocQA and the public NaVQA benchmarks. Furthermore, we successfully deployed Meta-Memory on real-world robotic platforms, demonstrating its practical utility in complex environments. Project page: https://itsbaymax.github.io/meta-memory.github.io/ .

Problem

• Enabling robots to effectively store observations as memories for spatial reasoning
• Developing mechanisms for efficient memory retrieval and integration in robotics
• Answering human natural language queries about spatial locations using semantic-spatial memories

Innovation

• LLM-driven agent constructs high-density memory representation
• Retrieves and integrates memories via semantic-spatial joint reasoning
• Enables robust spatial reasoning for natural language queries