preference learning

This paper addresses the challenge of aligning LLM-based evaluators with human preferences—characterized by miscalibration, sensitivity to scoring criteria, inherent biases, and output instability—by proposing a multi-judge learning framework. Methodologically, it introduces (1) a role-aware preference labeling approach that explicitly models the diversity of human judgments; (2) a unified aggregation framework for multi-judge outputs, incorporating dual-path aggregators—Generalized Additive Models (GAM) and Multi-Layer Perceptrons (MLP)—to enhance robustness; and (3) a score-conditioned judging mechanism that decouples absolute scoring from relative preference estimation. Empirically, the framework achieves significant improvements over baselines in reward modeling and model routing tasks. It demonstrates superior resilience to judge bias and output instability, yielding preference distributions closer to ground-truth human judgments. This work establishes a novel paradigm for trustworthy LLM evaluation.

To address the challenge of efficiently identifying preference-aligned solutions from the Pareto front in multi-objective optimization, this paper proposes a Bayesian preference-based active learning framework. The method models the decision maker’s implicit utility function using pairwise comparison feedback and employs an exploration-exploitation-balanced active sampling strategy, supporting both interactive and posterior usage modes. Its key contributions are: (i) the first systematic integration of Bayesian active learning into high-dimensional (up to nine objectives) multi-objective preference learning, substantially reducing query complexity; and (ii) robust convergence to high-satisfaction solutions with only a small number of pairwise comparisons across multiple benchmark problems. An open-source implementation is provided to facilitate practical adoption and reproducibility.

Existing AI preference alignment models neglect human cognitive mechanisms—such as heuristic simplification—resulting in cognitively implausible and poorly generalizable decision policies. Method: We propose an axiomatic, cognitively faithful decision-making framework: first performing pairwise comparisons over option attributes, then aggregating outcomes via fixed, interpretable rules. Our approach integrates cognitive decision theory with the Bradley–Terry model, constructing a structured information-processing pipeline grounded in empirically observed pairwise comparison data. Contribution/Results: Evaluated on kidney allocation, our model achieves state-of-the-art fidelity to human decisions while substantially improving interpretability and cross-task generalization—demonstrated via rigorous out-of-distribution validation. By grounding value alignment in empirically supported cognitive principles without sacrificing computational tractability, our framework offers a novel, principled pathway toward both cognitive plausibility and engineering feasibility in preference-aligned AI systems.

Quantifying user preferences remains challenging in multi-objective combinatorial optimization. Method: This paper proposes an interactive preference learning framework that efficiently infers a weighted objective function and generates high-quality solutions using minimal human pairwise comparisons. It introduces a constructive preference elicitation (CPE) approach integrating relaxed solution pool sampling, maximum-likelihood estimation under the Bradley–Terry model, and an ensemble-based active learning query strategy. Contribution/Results: Evaluated on PC configuration and multi-instance path planning tasks, the framework reduces required interactions by 37% on average compared to state-of-the-art CPE methods, while improving query efficiency and solution quality—measured by a 22% increase in Pareto front coverage. It establishes a new paradigm for preference-driven optimization that achieves both low interaction cost and high solution accuracy.

Existing preference datasets lack explicit modeling of human decision rationales, resulting in inefficient alignment learning, susceptibility to redundant expressions and hallucinations, and prohibitively high annotation costs. This paper proposes a data-centric rationale-augmented paradigm: (1) it is the first to systematically validate the efficacy of free large language models (LLMs) for self-generating decision rationales in preference learning; (2) it introduces a lightweight, general-purpose rationale injection framework that requires no additional human annotation or model fine-tuning and is compatible with mainstream algorithms such as DPO and KTO; and (3) it jointly optimizes rationale-guided contrastive learning and supervised fine-tuning. Experiments demonstrate substantial improvements in data efficiency and training convergence speed, consistent reductions in hallucination rates and redundant outputs across multiple benchmarks, and state-of-the-art performance gains.

This work addresses the critical challenge of accurately modeling preference functions that aggregate multidimensional criteria into holistic judgments in settings such as admissions and medical diagnosis. Departing from conventional assumptions of linearity or strong structural forms, the paper proposes the first robust nonparametric learning algorithm that achieves optimal performance without requiring any prior knowledge of the preference structure, assuming only monotonic non-decreasing behavior across each criterion. Theoretical analysis demonstrates the severe consequences of common model misspecifications, while experiments on both synthetic and real-world data confirm that the method maintains statistical efficiency under linear preferences and reliably recovers true evaluator preferences in general cases. Notably, the approach effectively uncovers key behavioral differences between human evaluators and large language models in their assessment strategies.

Current large language model alignment relies on ordinal human preferences (e.g., pairwise comparisons), yet theoretical analysis shows such preferences cannot uniquely identify the optimal policy—especially when trade-offs across orthogonal dimensions (e.g., factual accuracy vs. stylistic fluency) are required. Method: This work identifies this as a fundamental limitation and proposes cardinal human feedback, quantifying response quality differences via willingness-to-pay (WTP). We construct a high-fidelity cardinal dataset of 25K samples and integrate it into preference tuning via a novel cardinal loss function. Results: Experiments demonstrate consistent and significant gains over ordinal methods on benchmarks including Arena-Hard, enabling more precise identification of high-impact improvements and enhancing alignment efficiency and robustness. Core contribution: We establish the necessity of cardinal feedback for unambiguous model-level preference identification, thereby advancing beyond the conventional ordinal paradigm in preference learning.

To address the high annotation cost of human feedback in AI alignment and the low statistical efficiency and high redundancy of random pairwise sampling under existing Bradley–Terry models, this paper proposes Swiss InfoGain: a resource-aware preference sample selection method that integrates the Swiss-system tournament mechanism with mutual information gain-driven adaptive pairing. Grounded in game theory, statistical inference, and social choice theory, Swiss InfoGain dynamically prioritizes candidate pairs with maximal information content and highest uncertainty, substantially reducing labeling redundancy. Experiments demonstrate that, under constrained annotation budgets, Swiss InfoGain achieves significantly higher sample efficiency than baseline methods; in high-resource settings, it further improves final model alignment performance and robustness. The core contribution is the first integration of structured tournament mechanisms with information-theoretic criteria into a preference learning sampling framework.

This paper addresses the problem of learning individual heterogeneous preferences from partially observed choice behaviors. Conventional Bradley–Terry–Luce (BTL) models fail to capture preference heterogeneity and nonlinear user-item interactions. To overcome this, we propose a generalized BTL ranking model that represents users and items via low-dimensional latent features and models their interaction through a nonparametric preference function, yielding a score matrix. Methodologically, we design an indirect ℓ∞-regularization framework integrating sieve approximation, low-rank matrix estimation, and one-step Newton debiasing—enabling, for the first time, uncertainty quantification at both aggregate and individual ranking levels. We establish theoretical bounds on estimation error and empirically validate the method on synthetic and real-world datasets, demonstrating high-accuracy score prediction and reliable confidence assessment for rankings. The approach significantly enhances expressive power and statistical interpretability in preference modeling.

Traditional parametric discrete choice models struggle to capture complex decision rules under individual heterogeneity, particularly in modeling policy preferences. This study systematically evaluates four machine learning approaches—multinomial logistic regression, generalized additive models, Siamese neural networks, and Gaussian processes—across five behavioral and social science discrete choice tasks. Using both Monte Carlo simulations and real-world energy policy preference data, models are compared via Bayesian Information Criterion (BIC) and predictive accuracy. Results show that semi-parametric and non-parametric models consistently outperform parametric ones; increasing training sample size and choice rule determinism improves performance by 6%–96% and 0%–55%, respectively. On empirical data, the Siamese neural network achieves the best fit (BIC = 13.351). These findings underscore that model selection should be guided by task characteristics and highlight both the promise and limitations of data-driven methods in policy-oriented discrete choice modeling.