Janus: An Algorithm-Evaluator Co-Evolution Framework for LLM-Driven Discovery under Expensive Evaluation Budgets

2026-08-08

AI Summary

This work addresses the high cost of ground-truth evaluation—such as high-fidelity simulation or physical experimentation—that limits large language model (LLM)-driven program discovery in scientific and engineering domains. To overcome this bottleneck, the authors propose Janus, a novel framework that co-evolves target programs and executable agent evaluators. Janus leverages LLMs to generate task-specific evaluators and integrates region-conditioned composition, online credit assignment, and a closed-loop validation mechanism with real-world feedback. This approach efficiently guides the search process under label scarcity and distributional shift. Evaluated across five diverse tasks, Janus achieves 99% of baseline performance using only 40.9% of the ground-truth evaluations on average, substantially improving sample efficiency and the accuracy of candidate ranking.

Abstract

LLM-driven program discovery relies on rapid evaluator feedback, but many scientific and engineering tasks require high-fidelity simulations, hardware execution, or physical experiments, making each evaluation expensive. Cheap surrogate evaluators can reduce this cost, yet fixed surrogates are vulnerable to search-induced distribution shift and are difficult to fit reliably from sparse, search-biased labels. We introduce Janus, a framework that uses LLMs to co-evolve target programs and executable proxy evaluators. To address label scarcity, Janus leverages domain knowledge encoded in LLMs to generate task-specific evaluator programs and calibrates them using real outcomes. To mitigate distribution shift, Janus evolves evaluators alongside target programs, selects them using a promotion-aligned objective, and maintains region-conditioned portfolios with online credit updates. Because proxy predictions remain fallible, Janus uses them only to prioritize candidates and requires real validation before candidates can enter the target-program population or update the incumbent. Across five scientific and engineering design tasks, Janus achieves a larger area under the best-so-far improvement curve over the real-evaluation budget and higher final performance than a matched baseline that evolves only target programs. On average, Janus reaches 99/% of the baseline's final improvement with 59.1/% fewer real evaluations. Evolved proxy evaluators also rank promising candidates more accurately than their seed versions. Together, these results extend evaluator-guided LLM discovery from tasks with cheap, scalable feedback to scientific domains where trustworthy evaluation is scarce and expensive.

Problem

• expensive evaluation
• distribution shift
• surrogate evaluator
• LLM-driven discovery
• label scarcity

Innovation

• co-evolution
• surrogate evaluator
• distribution shift
• LLM-driven discovery
• expensive evaluation