Best-of-$infty$ -- Asymptotic Performance of Test-Time Compute

2025-09-25

AI Summary

While Best-of-N majority voting in large language models (LLMs) approaches optimal performance as N → ∞ (Best-of-∞), this limit is computationally infeasible. Method: We propose an efficient, budget-constrained approximation framework: (1) formalize a theoretical Best-of-∞ model to characterize its asymptotic performance bound; (2) design an answer-consistency–driven adaptive mechanism to dynamically select N per query under fixed computational budget; (3) extend to weighted multi-model ensembling, jointly optimizing model weights and per-model sampling sizes via mixed-integer linear programming (MILP). Results: Our method closely approximates Best-of-∞ performance within finite test budgets, improving inference efficiency by up to 2.3×. The weighted ensemble achieves consistent gains over the best individual model across multiple benchmarks, with average accuracy improvements of +4.7%. This work establishes the first theoretically grounded, scalable, and optimization-aware adaptive inference paradigm for LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We study best-of-$N$ for large language models (LLMs) where the selection is based on majority voting. In particular, we analyze the limit $N o infty$, which we denote as Best-of-$infty$. While this approach achieves impressive performance in the limit, it requires an infinite test-time budget. To address this, we propose an adaptive generation scheme that selects $N$ based on answer agreement, thereby efficiently allocating inference-time computation. Beyond adaptivity, we extend the framework to weighted ensembles of multiple LLMs, showing that such mixtures can outperform any individual model. The optimal ensemble weighting is formulated and efficiently computed as a mixed-integer linear program. Extensive experiments demonstrate the effectiveness of our approach.

Problem

• Analyzing majority voting performance for large language models with infinite test-time computation
• Proposing adaptive generation schemes to efficiently allocate inference-time computational resources
• Extending framework to weighted ensembles that outperform individual models via optimal weighting

Innovation

• Adaptive generation based on answer agreement
• Weighted ensembles of multiple LLMs
• Optimal weighting via mixed-integer linear program