Majority of the Bests: Improving Best-of-N via Bootstrapping

2025-11-23

AI Summary

Best-of-N (BoN) sampling fails in discrete-answer tasks when reward models are imperfect, as flawed reward signals lead to suboptimal candidate selection. Method: This paper proposes Majority-of-the-Bests (MoB), a robust alternative that bootstraps the BoN output distribution and selects the mode—i.e., the most frequently generated answer among top-k candidates—as the final prediction. Contribution/Results: We prove MoB’s statistical consistency under mild assumptions, establishing a novel sampling-based paradigm for candidate aggregation. Evaluated across 30 configurations—spanning 5 benchmarks, 3 foundation LLMs, and 2 reward models—MoB significantly outperforms BoN in 25 settings. Crucially, MoB does not require absolute accuracy of reward scores; it only assumes that relative ranking among candidates is reasonably reliable. This relaxation enhances both robustness and practical applicability in real-world reward modeling scenarios.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Sampling/Simulation-based Search
• Multiagent Systems: Mechanism Design

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Sampling multiple outputs from a Large Language Model (LLM) and selecting the most frequent (Self-consistency) or highest-scoring (Best-of-N) candidate is a popular approach to achieve higher accuracy in tasks with discrete final answers. Best-of-N (BoN) selects the output with the highest reward, and with perfect rewards, it often achieves near-perfect accuracy. With imperfect rewards from reward models, however, BoN fails to reliably find the correct answer and its performance degrades drastically. We consider the distribution of BoN's outputs and highlight that, although the correct answer does not usually have a probability close to one under imperfect rewards, it is often the most likely outcome. This suggests that the mode of this distribution can be more reliably correct than a sample from it. Based on this idea, we propose Majority-of-the-Bests (MoB), a novel selection mechanism that estimates the output distribution of BoN via bootstrapping and selects its mode. Experimental results across five benchmarks, three different base LLMs, and two reward models demonstrate consistent improvements over BoN in 25 out of 30 setups. We also provide theoretical results for the consistency of the bootstrapping. MoB serves as a simple, yet strong alternative to BoN and self-consistency, and more broadly, motivates further research in more nuanced selection mechanisms.

Problem

• Improving Best-of-N selection with imperfect reward models
• Estimating output distribution via bootstrapping for better accuracy
• Proposing Majority-of-the-Bests as alternative selection mechanism

Innovation

• Bootstrapping estimates Best-of-N output distribution
• Selects mode of distribution via Majority-of-the-Bests
• Improves reliability with imperfect reward models