When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger

2026-03-05

AI Summary

This work proposes Confidence-Weighted Preference Optimization (CW-PO), a framework designed to reduce reliance on costly human annotations or strong-model APIs for aligning large language models. CW-PO leverages weak language models to generate high-confidence preference samples and dynamically reweights these samples during training based on their confidence scores. Notably, the study reveals that high-confidence samples from weak models can yield superior alignment performance compared to using the full set of human-annotated data. Experimental results demonstrate that CW-PO, when combined with only 20% of the original human annotations, outperforms standard Direct Preference Optimization (DPO) trained on 100% annotated data, achieving both higher alignment efficiency and improved performance.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Preference alignment is an essential step in adapting large language models (LLMs) to human values, but existing approaches typically depend on costly human annotations or large-scale API-based models. We explore whether a weak LLM can instead act as an effective annotator. We surprisingly find that selecting only a subset of a weak LLM's highly confident samples leads to substantially better performance than using full human annotations. Building on this insight, we propose Confidence-Weighted Preference Optimization (CW-PO), a general framework that re-weights training samples by a weak LLM's confidence and can be applied across different preference optimization objectives. Notably, the model aligned by CW-PO with just 20% of human annotations outperforms the model trained with 100% of annotations under standard DPO. These results suggest that weak LLMs, when paired with confidence weighting, can dramatically reduce the cost of preference alignment while even outperforming methods trained on fully human-labeled data.

Problem

• preference alignment
• large language models
• human annotations
• weak LLMs
• cost reduction

Innovation

• Confidence-Weighted Preference Optimization
• weak LLMs
• preference alignment
• confidence weighting
• DPO