Leveraging Sparsity for Sample-Efficient Preference Learning: A Theoretical Perspective

2025-01-30

AI Summary

This paper addresses the sample efficiency challenge in high-dimensional sparse preference learning, where human annotation is costly and standard estimation rates Θ(d/n) are inadequate. We propose the Sparse Random Utility Model (SRUM), the first framework explicitly designed for sparsity in preference learning. We derive the tight minimax estimation rate Θ(k/n · log(d/k)) under SRUM and prove that an ℓ₁-regularized estimator achieves near-optimal rates under mild conditions—including restricted eigenvalue and compatibility assumptions on the Gram matrix. Our theoretical analysis establishes both upper and lower bounds on the minimax risk. Experiments on synthetic data and large language model alignment preference datasets demonstrate that our sparsity-aware method significantly reduces sample complexity and improves prediction accuracy. The work provides both theoretical guarantees and practical algorithms for low-resource preference modeling.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

This paper considers the sample-efficiency of preference learning, which models and predicts human choices based on comparative judgments. The minimax optimal estimation rate $Theta(d/n)$ in traditional estimation theory requires that the number of samples $n$ scales linearly with the dimensionality of the feature space $d$. However, the high dimensionality of the feature space and the high cost of collecting human-annotated data challenge the efficiency of traditional estimation methods. To remedy this, we leverage sparsity in the preference model and establish sharp estimation rates. We show that under the sparse random utility model, where the parameter of the reward function is $k$-sparse, the minimax optimal rate can be reduced to $Theta(k/n log(d/k))$. Furthermore, we analyze the $ell_{1}$-regularized estimator and show that it achieves near-optimal rate under mild assumptions on the Gram matrix. Experiments on synthetic data and LLM alignment data validate our theoretical findings, showing that sparsity-aware methods significantly reduce sample complexity and improve prediction accuracy.

Problem

• Sparse Preferences
• Learning Efficiency
• Prediction Accuracy

Innovation

• Sparse Preferences
• Efficient Learning
• Reduced Sample Complexity