Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions

2026-05-25

AI Summary

Existing embodied multimodal large language model (MLLM) agents struggle to capture users’ implicit intentions during long-term interactions and lack the capacity for continuous modeling of personalized context. This work proposes POLAR, a novel framework that, for the first time, integrates multimodal knowledge graphs with memory mechanisms to construct personalized contextual representations by fusing semantic and episodic memory. This representation supports retrieval and reasoning across multiple interaction sessions, enabling multi-hop inference and dynamic tracking of user states. Evaluated across diverse MLLM backbones and scenarios, POLAR significantly improves task success rates, particularly excelling in tasks requiring cross-interaction understanding and modeling of evolving contextual dynamics.

Abstract

Multimodal large language model (MLLM)-based embodied agents have shown strong potential for solving complex tasks in physical environments. However, personalized assistance requires more than following generic instruction or recognizing object categories. In real-world scenarios, the intended target is often specified only implicitly through prior interactions, requiring agents to leverage personalized context accumulated over time. In this work, we propose POLAR, a multiomodal memory-augmented framework for personalized embodied agents over long-term user interactions. POLAR organizes prior interactions into a multimodal knowledge graph that captures semantic memory for personalized context and visual concepts, and episodic memory for embodied experiences such as agent trajectories. To execute embodied tasks, POLAR retrieves relevant memories to interpret the current request and guide task execution. We evaluate POLAR across multiple MLLM backbones and diverse evaluation scenarios to study the role of memory in long-term personalization. Results show that the proposed memory mechanism consistently improves performance by enabling more effective use of information accumulated over prior interactions. The gains are especially pronounced when the agents are required to reason across multiple interactions, perform multi-hop inference, or tracking updates in user-specific context over time.

Problem

• personalization
• embodied agents
• multimodal large language models
• long-term interaction
• user context

Innovation

• personalized embodied agents
• multimodal memory
• knowledge graph
• long-term interaction
• multimodal large language models