Embodied Agents Meet Personalization: Exploring Memory Utilization for Personalized Assistance

2025-05-22

AI Summary

This work addresses the challenge of enabling embodied agents to leverage memory for understanding user-specific physical semantics (e.g., “my coffee cup”) and behavioral patterns (e.g., morning routines) to deliver dynamic, personalized assistance. To this end, we propose MEMENTO—a framework featuring a memory-augmented instruction parser and target localization module, coupled with a novel two-stage memory evaluation protocol that separately quantifies an agent’s capability in recognizing personalized object semantics and inferring regularized spatial locations. Experiments systematically reveal, for the first time, up to a 30.5% performance degradation in state-of-the-art large language models (e.g., GPT-4o) on multi-memory-dependent tasks, precisely identifying bottlenecks in memory utilization. Our contributions include a reproducible benchmark, diagnostic tools, and methodological foundations for developing personalized embodied intelligence.

Technology Category

• Intelligent Robots: Embodied AI
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Modeling other Agents

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Embodied agents empowered by large language models (LLMs) have shown strong performance in household object rearrangement tasks. However, these tasks primarily focus on single-turn interactions with simplified instructions, which do not truly reflect the challenges of providing meaningful assistance to users. To provide personalized assistance, embodied agents must understand the unique semantics that users assign to the physical world (e.g., favorite cup, breakfast routine) by leveraging prior interaction history to interpret dynamic, real-world instructions. Yet, the effectiveness of embodied agents in utilizing memory for personalized assistance remains largely underexplored. To address this gap, we present MEMENTO, a personalized embodied agent evaluation framework designed to comprehensively assess memory utilization capabilities to provide personalized assistance. Our framework consists of a two-stage memory evaluation process design that enables quantifying the impact of memory utilization on task performance. This process enables the evaluation of agents' understanding of personalized knowledge in object rearrangement tasks by focusing on its role in goal interpretation: (1) the ability to identify target objects based on personal meaning (object semantics), and (2) the ability to infer object-location configurations from consistent user patterns, such as routines (user patterns). Our experiments across various LLMs reveal significant limitations in memory utilization, with even frontier models like GPT-4o experiencing a 30.5% performance drop when required to reference multiple memories, particularly in tasks involving user patterns. These findings, along with our detailed analyses and case studies, provide valuable insights for future research in developing more effective personalized embodied agents. Project website: https://connoriginal.github.io/MEMENTO

Problem

• Assessing memory use in embodied agents for personalization
• Evaluating object semantics understanding in rearrangement tasks
• Measuring performance drop in models referencing multiple memories

Innovation

• MEMENTO framework evaluates memory utilization for personalization
• Two-stage process assesses object semantics and user patterns
• Reveals LLM limitations in multi-memory reference tasks