REAR: Test-time Preference Realignment through Reward Decomposition

2026-06-29

AI Summary

This work addresses the challenge of efficiently adapting large language models to diverse user preferences at test time, particularly in non-verifiable tasks where effective alignment mechanisms are lacking. The authors propose REAR, a novel framework that decomposes the reward function into problem-dependent and preference-dependent components, enabling efficient computation of test-time rewards. By integrating this reward with scalable decoding strategies such as Best-of-N sampling or tree search, REAR achieves dynamic, training-free alignment to user preferences. This approach circumvents the conventional reliance on annotated data and model retraining, significantly outperforming existing test-time baselines across multiple preference settings. Moreover, it demonstrates strong generalization to domains beyond text generation, including mathematical reasoning and vision tasks, highlighting its efficiency and flexibility.

Abstract

Aligning large language models (LLMs) with diverse user preferences is a critical yet challenging task. While post-training methods can adapt models to specific needs, they often require costly data curation and additional training. Test-time scaling (TTS) presents an efficient, training-free alternative, but its application has been largely limited to verifiable domains like mathematics and coding, where response correctness is easily judged. To extend TTS to preference alignment, we introduce a novel framework that models the task as a realignment problem, since the base model often fails to sufficiently align with the stated preference. Our key insight is to decompose the underlying reward function into two components: one related to the question and the other to preference information. This allows us to derive a REAlignment Reward (REAR) that selectively rescales the proportions of these two reward terms. We then show that REAR can be formulated as a linear combination of token-level policy log-probabilities, making it computationally efficient and easy to integrate with various TTS algorithms such as best-of-$N$ sampling and tree search. Experiments show that compared to other test-time baselines, REAR not only enables scalable test-time realignment for preference alignment tasks under diverse user requirements, but also generalizes to mathematical and visual tasks under appropriate preference settings.

Problem

• preference alignment
• test-time scaling
• large language models
• reward decomposition
• user preferences

Innovation

• test-time scaling
• reward decomposition
• preference alignment
• large language models
• REAR