TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization

2026-04-30

AI Summary

Traditional DPO methods rely solely on win/loss signals to model human preferences, neglecting the structural properties and uncertainty inherent in reasoning processes, which renders them fragile under noisy preference data. This work proposes TUR-DPO, the first approach to integrate lightweight reasoning topology and uncertainty estimation directly into the DPO objective. It introduces a weighted preference function that jointly accounts for semantic faithfulness, utility, and topological quality, enabling fine-grained alignment of reasoning trajectories without requiring reinforcement learning. Experimental results demonstrate that TUR-DPO significantly improves win rates, faithfulness, and calibration across diverse tasks—including mathematical reasoning, factual question answering, summarization, and dialogue—and exhibits robust performance in multimodal and long-context settings, consistently matching or surpassing PPO-based methods.

Abstract

Aligning large language models (LLMs) with human preferences is commonly done via reinforcement learning from human feedback (RLHF) with Proximal Policy Optimization (PPO) or, more simply, via Direct Preference Optimization (DPO). While DPO is stable and RL-free, it treats preferences as flat winner vs. loser signals and is sensitive to noisy or brittle preferences arising from fragile chains of thought. We propose TUR-DPO, a topology- and uncertainty-aware variant of DPO that rewards how answers are derived, not only what they say, by eliciting lightweight reasoning topologies and combining semantic faithfulness, utility, and topology quality into a calibrated uncertainty signal. A small learnable reward is factorized over these signals and incorporated into an uncertainty-weighted DPO objective that remains RL-free and relies only on a fixed or moving reference policy. Empirically, across open 7-8B models and benchmarks spanning mathematical reasoning, factual question answering, summarization, and helpful/harmless dialogue, TUR-DPO improves judge win-rates, faithfulness, and calibration relative to DPO while preserving training simplicity and avoiding online rollouts. We further observe consistent gains in multimodal and long-context settings, and show that TUR-DPO matches or exceeds PPO on reasoning-centric tasks while maintaining operational simplicity.

Problem

• Direct Preference Optimization
• reasoning topology
• preference noise
• uncertainty calibration
• LLM alignment

Innovation

• Topology-aware
• Uncertainty-aware
• Direct Preference Optimization
• Reasoning Chains
• Calibrated Reward