A Markov Chain Approach to Preference Alignment

2026-06-21

AI Summary

Existing alignment methods often reduce pairwise human preferences to scalar rewards, resulting in significant information loss. This work proposes MCHF, a novel approach that directly embeds pairwise preferences into the transition kernel of a Markov chain, enabling iterative sampling for aligning generative models without explicit reward modeling. MCHF unifies RLHF, NLHF, and the Markov framework, revealing their intrinsic connections under non-transitive preferences. Theoretical analysis shows that MCHF converges geometrically to a stationary distribution, with the convergence rate governed by the degree of preference non-transitivity: it recovers the RLHF solution at the first iteration and progressively captures higher-order preference structures in subsequent steps.

Abstract

We propose Markov Chain from Human Feedback (MCHF), an elementary approach for aligning generative models from pairwise human preferences. Unlike Reinforcement Learning from Human Feedback (RLHF), which reduces comparisons to a scalar reward, and Nash Learning from Human Feedback (NLHF), which preserves pairwise utilities through a KL-regularized minimax optimization, MCHF uses pairwise preferences directly to define a transition mechanism over model outputs. Given a pairwise utility $U(x,y)$, which quantifies human preference for $y$ over $x$, and a reference probability distribution $μ_{\mathsf{ref}}$, we define a Markov kernel $\mathsf{P}(x, dy)\propto \exp(U(x,y))μ_{\mathsf{ref}}(dy)$, and take the Markov chain starting from $μ_{\mathsf{ref}}$ as an iterative alignment procedure. We show that MCHF converges geometrically fast to the stationary distribution, with a convergence rate governed by the seminorm $\|U\|_\oplus=\inf_{g,f\in L^\infty(μ_{\mathsf{ref}})}\|U-g\oplus f\|_\infty$, which quantifies the non-transitive structure of the pairwise utility. We further show that a mirror-descent algorithm for NLHF satisfies an analogous structure-adaptive convergence guarantee. Finally, through a perturbation analysis, we prove that when $\|U\|_\oplus$ is small, MCHF and NLHF agree up to first order around an RLHF solution, which yields a unified view of reward-based, game-theoretic, and Markovian approaches to alignment. In particular, for two natural algorithms that converge to the MCHF/NLHF equilibria, we show that the first step of MCHF and NLHF recovers the RLHF solution based on the column-sum reward $\hat{f}(y)=\int μ_{\mathsf{ref}}(dx) U(x, y)$, and starting from the second iteration, both algorithms incorporate the same linear functional of the residual $U-(-\hat f)\oplus \hat f$, which captures the non-transitive structure of the pairwise utility $U$.

Problem

• preference alignment
• pairwise preferences
• non-transitive structure
• generative models
• human feedback

Innovation

• Markov Chain from Human Feedback
• preference alignment
• pairwise utility
• non-transitive structure
• geometric convergence