KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systems

2024-09-23

arXiv.org

AI Summary

To address planning failures and low efficiency in embodied AI for long-horizon household tasks—caused by insufficient contextual memory—this paper introduces KARMA, a novel dual-memory architecture that jointly models long-term memory (via 3D scene graph representation) and short-term memory (via dynamic object-state tracking), augmented with an adaptive memory replacement mechanism. KARMA enhances embodied planning capabilities of large language models (LLMs) through memory-augmented prompting and supports plug-and-play deployment on robotic platforms. Evaluated in AI2-THOR, KARMA achieves a 1.3× improvement in success rate on composite tasks and a 2.3× gain on complex tasks, while execution efficiency improves by 3.4× and 62.7×, respectively. Crucially, the system successfully transfers to a real-world mobile manipulator, demonstrating strong sim-to-real generalization.

Technology Category

• Intelligent Robots: Embodied AI
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Embodied AI agents responsible for executing interconnected, long-sequence household tasks often face difficulties with in-context memory, leading to inefficiencies and errors in task execution. To address this issue, we introduce KARMA, an innovative memory system that integrates long-term and short-term memory modules, enhancing large language models (LLMs) for planning in embodied agents through memory-augmented prompting. KARMA distinguishes between long-term and short-term memory, with long-term memory capturing comprehensive 3D scene graphs as representations of the environment, while short-term memory dynamically records changes in objects' positions and states. This dual-memory structure allows agents to retrieve relevant past scene experiences, thereby improving the accuracy and efficiency of task planning. Short-term memory employs strategies for effective and adaptive memory replacement, ensuring the retention of critical information while discarding less pertinent data. Compared to state-of-the-art embodied agents enhanced with memory, our memory-augmented embodied AI agent improves success rates by 1.3x and 2.3x in Composite Tasks and Complex Tasks within the AI2-THOR simulator, respectively, and enhances task execution efficiency by 3.4x and 62.7x. Furthermore, we demonstrate that KARMA's plug-and-play capability allows for seamless deployment on real-world robotic systems, such as mobile manipulation platforms.Through this plug-and-play memory system, KARMA significantly enhances the ability of embodied agents to generate coherent and contextually appropriate plans, making the execution of complex household tasks more efficient. The experimental videos from the work can be found at https://youtu.be/4BT7fnw9ehs. Our code is available at https://github.com/WZX0Swarm0Robotics/KARMA/tree/master.

Problem

• Enhancing embodied AI agents' memory for household tasks
• Integrating long-and-short term memory for task planning
• Improving efficiency and accuracy in task execution

Innovation

• Integrates long-term and short-term memory modules
• Uses 3D scene graphs for environment representation
• Enables plug-and-play deployment on robotic systems