Learning Personalized Agents from Human Feedback

2026-02-17

AI Summary

Existing AI agents struggle to continuously align with users’ dynamically evolving personalized preferences, particularly in scenarios involving new users or preference drift. To address this limitation, this work proposes the PAHF framework, which uniquely integrates explicit user memory with a dual-channel feedback mechanism. Through an iterative “clarify–retrieve–feedback” loop, PAHF enables online preference learning and rapid adaptation from scratch. This approach overcomes the constraints of static datasets and implicit modeling, significantly outperforming memory-less and single-feedback baselines on embodied manipulation and online shopping benchmarks. The method substantially reduces initial personalization error and effectively supports cross-task preference transfer.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Modern AI agents are powerful but often fail to align with the idiosyncratic, evolving preferences of individual users. Prior approaches typically rely on static datasets, either training implicit preference models on interaction history or encoding user profiles in external memory. However, these approaches struggle with new users and with preferences that change over time. We introduce Personalized Agents from Human Feedback (PAHF), a framework for continual personalization in which agents learn online from live interaction using explicit per-user memory. PAHF operationalizes a three-step loop: (1) seeking pre-action clarification to resolve ambiguity, (2) grounding actions in preferences retrieved from memory, and (3) integrating post-action feedback to update memory when preferences drift. To evaluate this capability, we develop a four-phase protocol and two benchmarks in embodied manipulation and online shopping. These benchmarks quantify an agent's ability to learn initial preferences from scratch and subsequently adapt to persona shifts. Our theoretical analysis and empirical results show that integrating explicit memory with dual feedback channels is critical: PAHF learns substantially faster and consistently outperforms both no-memory and single-channel baselines, reducing initial personalization error and enabling rapid adaptation to preference shifts.

Problem

• personalization
• human feedback
• preference drift
• AI alignment
• user modeling

Innovation

• personalized agents
• human feedback
• explicit memory
• continual learning
• preference adaptation