Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies

2026-08-12

AI Summary

This work addresses the limitation of large language models in mathematical and scientific discovery tasks, where reliance on a single “best-guess” output often fails to capture the diverse solution space, thereby constraining pass@k performance. The paper introduces, for the first time, a systematic application of Evolution Strategies to test-time post-training of large language models. By injecting random perturbations into the model’s weight space, this approach preserves solution diversity without inducing output distribution collapse. Empirical results demonstrate that the method substantially improves both pass@k scores and coverage of the solution space on standard mathematical benchmarks, outperforming existing reinforcement learning–based post-training techniques.

Abstract

Large Language Models (LLMs) are increasingly deployed in discovery domains such as math and science. The usual approach is to present the problem to the model and use its answer as the proposed solution. However, beyond this best guess, discovery can be enhanced by increasing test-time compute. In a process called pass@k, the model is allowed to explore the solution space and generate diverse candidate solutions. Unfortunately, the standard approach to post-training LLMs through Reinforcement Learning (RL) may limit pass@k: the model's output distribution narrows around high-reward outputs, causing the solution coverage to collapse. The alternative is to use Evolution Strategies (ES), a population-based, gradient-free post-training method that optimizes directly in weight space through random perturbations. As this paper shows, ES achieves consistently higher pass@k than RL and produces a broader output distribution with greater solution coverage. This coverage in turn makes it possible to achieve better results in e.g. standard math benchmarks. Thus, ES provides a better foundation for post-training in discovery problems and other domains where diverse solution coverage is critical.

Problem

• Large Language Models
• solution coverage
• pass@k
• discovery domains
• output distribution collapse

Innovation

• Evolution Strategies
• pass@k
• solution coverage
• Large Language Models
• post-training