Neural Nonmyopic Bayesian Optimization in Dynamic Cost Settings

2026-01-10

arXiv.org

AI Summary

This work addresses the limitations of traditional Bayesian optimization in dynamic, history-dependent cost environments, where myopic strategies hinder effective long-horizon planning. The authors propose LookaHES, a novel framework that integrates neural policies—including large language models—into non-myopic Bayesian optimization for the first time. By combining multi-step H-entropy search, path sampling, and a constraint-aware rollout mechanism, LookaHES enables efficient exploration of structured combinatorial action spaces while naturally incorporating domain-specific constraints. The method supports decision-making over horizons exceeding twenty steps under dynamic costs and demonstrates significant performance gains over both myopic and existing non-myopic approaches across nine synthetic benchmarks and two real-world tasks: NASA nighttime light geospatial optimization and constrained protein sequence design.

Technology Category

• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Bayesian optimization (BO) is a common framework for optimizing black-box functions, yet most existing methods assume static query costs and rely on myopic acquisition strategies. We introduce LookaHES, a nonmyopic BO framework designed for dynamic, history-dependent cost environments, where evaluation costs vary with prior actions, such as travel distance in spatial tasks or edit distance in sequence design. LookaHES combines a multi-step variant of $H$-Entropy Search with pathwise sampling and neural policy optimization, enabling long-horizon planning beyond twenty steps without the exponential complexity of existing nonmyopic methods. The key innovation is the integration of neural policies, including large language models, to effectively navigate structured, combinatorial action spaces such as protein sequences. These policies amortize lookahead planning and can be integrated with domain-specific constraints during rollout. Empirically, LookaHES outperforms strong myopic and nonmyopic baselines across nine synthetic benchmarks from two to eight dimensions and two real-world tasks: geospatial optimization using NASA night-light imagery and protein sequence design with constrained token-level edits. In short, LookaHES provides a general, scalable, and cost-aware solution for robust long-horizon optimization in complex decision spaces, which makes it a useful tool for researchers in machine learning, statistics, and applied domains. Our implementation is available at https://github.com/sangttruong/nonmyopia.

Problem

• Bayesian optimization
• dynamic cost
• nonmyopic planning
• history-dependent cost
• black-box optimization

Innovation

• nonmyopic Bayesian optimization
• neural policy
• dynamic cost
• long-horizon planning
• large language models