UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignment

2026-02-10

AI Summary

This work addresses the challenge of multi-objective alignment in autoregressive reward modeling, where existing approaches often suffer from entangled preference representations or insufficient interaction due to parameter or feature separation across objectives. To overcome this, the authors propose UniARM, a unified autoregressive reward model that jointly models all preference dimensions within a shared parameter space. Central to UniARM is a novel Mixture-of-Shared-Low-Rank Adaptation (MoSLoRA) architecture: it first extracts preference-agnostic shared features and then applies disentangled modulation via mixture preference vectors, enabling precise and controllable trade-offs among multiple objectives at inference time. UniARM is the first framework to support multi-objective alignment within a single model while maintaining compatibility with frozen large language models, thereby facilitating efficient deployment on large-scale systems and significantly enhancing controllability and practical utility.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Humans and AI: Learning Human Values and Preferences
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Multi-objective alignment aims to align LLM responses with multiple human preference objectives. Among existing methods, guiding the generation of frozen LLMs through autoregressive reward models (ARMs) to accomplish multi-objective test-time alignment is a low-cost solution. However, these methods typically rely on independent parameters for each preference objective, either by training ARMs independently across preference dimensions, which neglects interactions among preference features, or by training a single ARM with separate feature extraction modules for each preference, which can cause feature entanglement. Both strategies can result in misalignment between generated outputs and user preferences. To address this limitation, we propose Preference-Modulated \&Shared Low-Rank Adaptation (MoSLoRA) for ARM training, which first extracts shared features via a preference-agnostic module and then applies affine transformations to shared features via a preference modulation module conditioned on mixed preference vectors. This design mitigates feature entanglement and enables precise control over preference trade-offs during inference. Building on this, we introduce the Unified Autoregressive Reward Model (UniARM), a novel framework for multi-objective test-time alignment. UniARM jointly models all preference dimensions in a single parameter space, eliminating the need for independent parameters for each preference objective. es on larger-scale LLMs, enhancing its practical usability.

Problem

• multi-objective alignment
• autoregressive reward model
• feature entanglement
• test-time alignment
• preference modeling

Innovation

• Unified Autoregressive Reward Model
• Multi-objective Alignment
• Preference Modulation
• Shared Low-Rank Adaptation
• Test-Time Alignment