Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations

2026-08-14

AI Summary

This study addresses the inefficiency of fixed sampling in large language model evaluation, which often leads to resource wastage due to the absence of adaptive stopping mechanisms. We propose OptStop, a framework introducing a novel calibration-free hierarchical Bayesian adaptive stopping strategy that formulates evaluation as a sequential measurement problem. This approach dynamically allocates samples based on uncertainty quantification while incorporating a zero-performance safety fallback. Extensive experiments across nine validation settings demonstrate that OptStop achieves accuracy equivalent to exhaustive evaluation while reducing the required number of trials by 57% to 97%. These results indicate substantial improvements in both evaluation efficiency and computational resource utilization, offering a robust solution for cost-effective LLM benchmarking without compromising assessment reliability.

Abstract

LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise. We introduce optstop, a precision-based adaptive stopping framework that treats evaluation as a sequential measurement problem: keep sampling where uncertainty remains high, and stop where estimates are precise or stable enough. The framework builds on hierarchical Bayesian inference, supports binary, ordinal, and continuous outcomes, and keeps every benchmark item eligible for sampling, without requiring a calibrated item bank. It runs live or retrospectively, and includes a safeguard that samples more cautiously as measured performance approaches zero, where rare successes matter most. In an illustrative 200-item, 10-epoch evaluation, it removes 57%-97% of planned trials across nine validation settings, with overall conclusions equivalent to the full run. These results show that LLM evaluation compute can be allocated by uncertainty rather than by fixed repetition counts, with the magnitude of savings depending on evaluation design.

Problem

• LLM evaluation
• fixed sampling budget
• adaptive stopping
• computational efficiency

Innovation

• Bayesian Optimal Stopping
• Adaptive Evaluation Framework
• Hierarchical Bayesian Inference
• Uncertainty-based Sampling
• LLM Evaluation Efficiency