Learning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning

2026-08-10

AI Summary

This work addresses the inefficiency of generic user preference summaries, which often contain substantial task-irrelevant information that consumes precious context capacity and induces cross-task interference, while handcrafted task-specific views suffer from poor scalability. To overcome these limitations, the authors propose AlignXada, a novel framework that, for the first time, enables fine-tuning-free, task-adaptive preference summarization. AlignXada employs verbal reinforcement learning to train a meta-learner that iteratively refines reusable textual distillation strategies, compressing generic preferences into lightweight, task-relevant representations. Extensive experiments across 39 combinations of 13 diverse tasks and 3 large language models demonstrate that AlignXada improves average performance by 3.82 points—benefiting 33 out of 39 settings—while retaining only 22.8% of the original tokens and outperforming retrieval-augmented generation (RAG) baselines in 36 combinations.

Abstract

Natural language user preferences provide an interpretable interface for LLM personalization. However, universal preference summaries often contain information irrelevant to a particular downstream task. Directly supplying the full preference summary therefore wastes context capacity and introduces cross-task distraction, while manually designing task-specific preference views is difficult to scale. In this work, we study \emph{task-specific preference adaptation}: given a universal user preference summary and a downstream task, derive a task-conditioned representation that preserves sufficient decision-relevant evidence while removing redundant context. To this end, we propose \textsc{AlignXada}, a training-free meta-learning framework that induces reusable textual refinement policies for adapting universal preference summaries to task-specific ones. The refinement policy is iteratively optimized by a meta learner through verbal reinforcement learning. Across 13 tasks and three downstream models (39 task--model cells), \textsc{AlignXada} achieves an average gain of 3.82 points, improving 33 cells while retaining only 22.8\% of the original profile tokens and outperforming RAG in 36 cells. An extended faithfulness analysis further shows that the refined profiles remain largely grounded in the source preferences while preserving task-relevant personalization signals, suggesting that profile-side adaptation serves as a practical complement to universal memory construction for lifelong personalized agents.

Problem

• preference adaptation
• large language model personalization
• task-specific preference
• verbal reinforcement learning
• context efficiency

Innovation

• preference adaptation
• verbal reinforcement learning
• task-specific personalization
• training-free meta-learning
• context compression