PAPA: Online Personalized Active Preference Alignment

2026-07-01

AI Summary

This work addresses the challenge of aligning diffusion models with users’ dynamically revealed preferences in personalized recommendation scenarios where large-scale preference data is unavailable. The authors propose an online preference alignment framework that eliminates the need for a parameterized reward model. By leveraging a variational inference–driven active learning mechanism, the method directly optimizes the diffusion model using real-time user feedback and introduces EPAPA, a computationally efficient fine-tuning strategy. Departing from conventional paradigms that rely on offline preference datasets to train reward models, the proposed approach significantly improves sample efficiency and reduces computational overhead across diverse conditional generation and fine-grained alignment tasks, offering both practical utility and deployment feasibility.

Abstract

Diffusion models are highly effective at modeling complex data distributions, including images and text. However, in applications like personalized recommender systems, the objective often shifts to modeling specific regions of the distribution that maximize user preferences-initially unknown but gradually uncovered through interactive feedback. This can naturally be framed as a reinforcement learning problem, where the goal is to fine-tune a diffusion model to maximize a reward function based on preferences. However, the main challenge lies in learning a parameterized reward model, which typically requires large-scale preference data-something that is often not feasible in practice. In this work, we introduce Personalized Active Preference Alignment PAPA, a novel method that bypasses the requirement for a parametrized reward model by directly optimizing the diffusion model using real-time user feedback. PAPA enables feedback-efficient preference alignment, drawing inspiration from the variational inference framework. We demonstrate PAPA's effectiveness through extensive experiments and ablation studies across diverse class-conditioned and fine-grained alignment tasks. Additionally, based on theoretical insights, we propose an enhanced fine-tuning strategy, referred to as EPAPA, that requires less computational budget and accelerates the fine-tuning process, further boosting PAPA's suitability for real-world deployment. Our code is made publicly available at https://github.com/NasikNafi/papa.

Problem

• personalized preference alignment
• diffusion models
• reinforcement learning
• active feedback
• reward modeling

Innovation

• diffusion models
• preference alignment
• active learning
• reinforcement learning
• personalized recommendation