Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization

2026-08-10

AI Summary

This work addresses a key limitation of standard Direct Preference Optimization (DPO), which treats all tokens uniformly despite their varying contributions to preference signals. To overcome this, the authors propose a dynamic token-level KL regularization mechanism that integrates a lightweight calibration network into the DPO framework. This network leverages the model’s internal implicit reward signals to continuously assess each token’s contribution strength and confidence during training, adaptively adjusting its credit weight without requiring any external model. Evaluated on AlpacaEval 2 and Arena-Hard, the method achieves performance gains of 9.8 and 12.2 points, respectively, significantly outperforming standard DPO. Notably, it enables the first self-evolving, dynamic calibration of token credit throughout the training process.

Abstract

Direct Preference Optimization (DPO) aggregates token-level log-probability ratios via uniform summation, implicitly treating all tokens as contributing equally to the preference signal. However, the contribution of individual tokens to the preference signal varies. We introduce token credit, which modulates each token's KL regularization based on its contribution to the preference outcome. We derive that effective token credit is proportional to the magnitude of each token's implicit reward, and observe that this quantity evolves substantially during training. This implies that static token credit becomes increasingly misaligned as training progresses. In this work, we propose Se-DPO (Self-Evolving Token Credit for DPO), a live mechanism that derives token credit from the model's own evolving internal signals during DPO training. Since the reward signal varies in reliability across positions, Se-DPO calibrates token credit based on both the strength and the confidence of each token's contribution. Se-DPO requires no external models, adding only a lightweight calibration network with minimal computational overhead. Experiments show that Se-DPO improves over DPO by up to 9.8 points on AlpacaEval~2 and 12.2 points on Arena-Hard.

Problem

• Direct Preference Optimization
• token credit
• preference signal
• KL regularization
• reward reliability

Innovation

• token credit
• Direct Preference Optimization
• self-evolving mechanism
• implicit reward
• preference alignment