Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings

2026-06-17

AI Summary

This study investigates whether sequential preference optimization leads to uniform forgetting of previously learned preferences and how this phenomenon is influenced by the relationships among preference objectives. Using Llama-3.1-8B-Instruct with LoRA adapters, the authors apply Direct Preference Optimization (DPO) sequentially across four distinct preference settings, employing length-normalized margins and quartile-based decomposition for fine-grained analysis alongside gradient diagnostics. The findings reveal that sequential DPO does not induce uniform forgetting; instead, it exhibits diverse behaviors ranging from degradation and stability to positive transfer. Objective compatibility and signal strength emerge as key determinants of these dynamics. High-confidence preference pairs can either improve or deteriorate across stages, and inter-stage gradients are nearly orthogonal, suggesting that gradient interference is not the primary cause of forgetting. These insights offer new design principles for multi-objective alignment.

Abstract

Aligning language models with human preferences often requires optimising multiple behavioural objectives. A practical approach is to apply these objectives sequentially using preference optimisation methods such as Direct Preference Optimisation (DPO), but it remains unclear whether later training uniformly degrades preferences learned earlier or whether the effect depends on the relationship between objectives. We study sequential DPO across four preference settings covering distributional conflict, multi-attribute interaction, strong safety signal, and compatible response-quality objectives. Using Llama-3.1-8B-Instruct with LoRA adapters, we evaluate all objectives after every stage with a fixed base-model reference. We find that sequential DPO does not produce a single forgetting pattern; preference change ranges from partial degradation to stability, pair-level redistribution, or positive transfer depending on objective relationship, signal strength, and training order. Pair-level analysis using length-normalised policy margins shows that aggregate metrics can mask heterogeneous changes across preference pairs, whereas quartile decomposition reveals that high-confidence pairs can either degrade or improve depending on the setting. Mechanistic diagnostics show that Stage~2 gradients and adapter updates are near-orthogonal to the previous objective across all settings, providing little evidence that direct gradient opposition is the primary driver. These findings suggest that future sequential alignment pipelines should account for objective compatibility and signal strength, rather than assuming that later objectives affect earlier preferences uniformly.

Problem

• sequential alignment
• preference forgetting
• Direct Preference Optimization
• objective compatibility
• multi-objective learning

Innovation

• Sequential Direct Preference Optimization
• Preference Forgetting
• Objective Compatibility
• Pair-level Analysis
• Gradient Orthogonality