STAGE: Controlled Objective Admission for Multi-Preference LLM Alignment

2026-08-17

AI Summary

This study addresses the lack of temporal control over dimension-specific optimization in scalarization methods for multi-preference alignment. We propose STAGE, a stability-guided controller that treats objective admission timing as an explicit variable. By leveraging cumulative retention and gating mechanisms, STAGE dynamically regulates the admission schedule and weights of preference dimensions, while incorporating a difficulty-ranked probing strategy to enable adaptive optimization. Extensive experiments across 15 training preferences and 16 benchmarks demonstrate that STAGE significantly outperforms baselines, including synchronous scalarization and shared-budget approaches, in average performance. These results confirm its effectiveness in enhancing multi-preference alignment for large language models by precisely managing when and how distinct preference objectives are integrated during training.

Abstract

Multi-preference alignment is often framed as scalarization: combine reward dimensions, then optimize. This leaves a temporal decision underspecified: when should each preference dimension enter policy optimization? We propose \methodname, a stability-guided active-set controller for controlled objective admission. \methodname starts from a small active set, retains admitted objectives, and expands when reward-deviation gates indicate low recent deviation or a patience budget is exhausted. A probing phase estimates a hard-to-easy order, and adaptive weighting emphasizes underperforming active dimensions. Automatic evaluations with 15 training preferences and 16 held-out benchmark columns show that \methodname obtains higher averages than simultaneous scalarization and shared-budget adapted baselines. Component ablations and expansion dynamics further support cumulative retention, gated admission, and probing-derived ordering as useful design choices in this setting. These results position objective-entry timing as a concrete control variable in reward-vector RLHF.

Problem

• Multi-preference alignment
• Objective admission timing
• Scalarization
• RLHF

Innovation

• Controlled Objective Admission
• Stability-guided Active-set Controller
• Reward-deviation Gates
• Hard-to-easy Ordering
• Multi-preference Alignment