Latent Adversarial Regularization for Offline Preference Optimization

2026-01-29

AI Summary

This work addresses the instability in current language model preference optimization methods, which rely on token-level regularization and fail to capture semantic or behavioral similarity. To overcome this limitation, the authors propose a latent-space adversarial regularization approach that, for the first time, integrates adversarial training into offline preference optimization. By minimizing the distributional discrepancy between the policy and reference models in the latent space, the method constructs a regularizer that does not require explicit density estimation. This approach effectively mitigates the semantic shortcomings of token-level regularization, significantly enhancing robustness to distributional shift and noisy feedback. It consistently yields performance improvements across diverse model architectures and tasks while introducing only minimal computational overhead.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Learning from human feedback typically relies on preference optimization that constrains policy updates through token-level regularization. However, preference optimization for language models is particularly challenging because token-space similarity does not imply semantic or behavioral similarity. To address this challenge, we leverage latent-space regularization for language model preference optimization. We introduce GANPO, which achieves latent-space regularization by penalizing divergence between the internal representations of a policy model and a reference model. Given that latent representations are not associated with explicit probability densities, we adopt an adversarial approach inspired by GANs to minimize latent-space divergence. We integrate GANPO as a regularizer into existing offline preference optimization objectives. Experiments across multiple model architectures and tasks show consistent improvements from latent-space regularization. Further, by comparing GANPO-induced inferential biases with those from token-level regularization, we find that GANPO provides more robust structural feedback under distributional shift and noise while maintaining comparable downstream performance with minor computational overhead.

Problem

• preference optimization
• language models
• latent space
• offline learning
• human feedback

Innovation

• latent-space regularization
• preference optimization
• adversarial training
• offline reinforcement learning
• language models