NextQuill: Causal Preference Modeling for Enhancing LLM Personalization

2025-06-03

AI Summary

Existing LLM personalization methods struggle to disentangle users’ genuine preferences from statistical biases in behavioral data, resulting in superficial alignment. To address this, we propose the first causal preference modeling framework for LLM personalization: treating user interaction history as a causal factor, we formally define and estimate token-level Preference Causal Effect (PCE). Our method introduces a dual-path alignment mechanism—internal PCE matching within the model and selective fitting of preference-carrying tokens—enabled by an end-to-end optimization pipeline integrating do-calculus, counterfactual estimation, preference-aware masking, and causal effect distillation. Extensive experiments on multiple personalized generation benchmarks demonstrate significant improvements over state-of-the-art methods. The code is publicly released, empirically validating both the effectiveness and generalizability of the causal paradigm for LLM personalization.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Personalizing large language models (LLMs) for individual users has become increasingly important as they are progressively integrated into real-world applications to support users' daily lives. However, existing personalization approaches often fail to distinguish which components of model predictions and training data truly reflect user preferences, leading to superficial personalization alignment. In this paper, we introduce NextQuill, a novel LLM personalization alignment framework grounded in causal preference modeling. We approach personalization from a causal perspective, treating both model predictions and ground-truth data generation as outcomes influenced by user preferences, along with other factors. We define the true preference effect as the causal impact of user history (which reflects preferences) on each token prediction or data generation instance, estimated through causal intervention techniques. Building on this insight, NextQuill introduces two complementary alignment strategies: (1) aligning model-internal causal preference effects on predictions with those reflected in ground-truth data, rather than indiscriminately fitting predictions, and (2) focusing on fitting preference-bearing tokens identified via ground-truth data preference effects, rather than treating all tokens uniformly. By integrating these strategies, NextQuill shifts the alignment process toward learning from causal preference effects, facilitating more effective and personalized adaptation. Experiments across multiple personalization benchmarks demonstrate that NextQuill significantly improves personalization quality, offering a principled, causal foundation for LLM personalization. Our codes are available on https://github.com/juntaoyou/NextQuill.

Problem

• Distinguishing true user preferences in LLM predictions and training data
• Aligning model predictions with causal preference effects in data
• Improving personalization quality through causal preference modeling techniques

Innovation

• Causal preference modeling for LLM personalization
• Aligning model-internal causal preference effects
• Focusing on fitting preference-bearing tokens