BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks

2026-05-16

AI Summary

Optimizing training and inference configurations for large language models (LLMs) has long lacked a reproducible black-box optimization (BBO) benchmark due to high costs, significant noise, and the absence of derivative information. This work proposes BoLT, the first open-source BBO benchmark tailored for LLMs, built upon thousands of real-world experiments and distilled into lightweight surrogate models. BoLT systematically incorporates practical challenges such as high dimensionality, multi-fidelity evaluations, multi-objective trade-offs, and heteroscedastic noise. The benchmark enables efficient, low-cost algorithm evaluation, and experimental results across diverse BBO methods reveal that certain Bayesian optimization approaches outperform others while also exposing limitations of current methods in modern LLM optimization scenarios.

Abstract

Optimization of LLM training and inference configurations, such as hyperparameters, data mixtures, and prompts, is critical to performance, but it is often approached heuristically in practice, leading to potentially suboptimal outcomes. By framing them as noisy, expensive, and derivative-free optimization problems, Bayesian optimization (BO) and other black-box optimization (BBO) methods offer a promising yet underexplored direction for principled, sample-efficient methods. However, LLM training and inference costs are prohibitively high for most of the BBO research community, and new methods are often only evaluated on synthetic test functions and small-scale datasets that fail to capture the challenges of modern LLM optimization problems. This impedes the development of BBO methods and makes it difficult to assess their effectiveness on modern LLM tasks. We introduce BoLT, the first LLM-centric benchmark that democratizes LLM research for the BBO community. BoLT is released at https://github.com/chewwt/bolt. BoLT covers broad and well-motivated LLM optimization problems, involving multi-fidelity, multi-objective, heteroscedastic noise, and high-dimensional search spaces. Each problem in BoLT is grounded in real experimental data and made fully reproducible and accessible through lightweight surrogate models fitted to the results of thousands of real LLM experiments. We benchmark BoLT against an extensive range of BO and BBO methods, showing that selected BO methods consistently outperform others across tasks and highlighting gaps in existing BBO methods on LLM tasks, underscoring the need to modernize benchmarks for the BBO community.

Problem

• black-box optimization
• large language models
• expensive optimization
• benchmarking
• Bayesian optimization

Innovation

• Black-box Optimization
• Bayesian Optimization
• Large Language Models
• Surrogate Models
• Benchmark