Approximation of differential entropy in Bayesian optimal experimental design

2025-10-01

AI Summary

In Bayesian optimal experimental design, evaluating the expected information gain (EIG) is computationally prohibitive due to the intractability of the marginal likelihood’s differential entropy. When the likelihood’s differential entropy is design-independent or analytically available, entropy estimation of the evidence density becomes the dominant bottleneck. To address this, we propose a novel strategy that relies only on weak smoothness assumptions on the forward model, constructing a surrogate for the evidence density via Monte Carlo or quasi-Monte Carlo sampling—thereby avoiding conventional strong regularity requirements. We establish theoretical guarantees showing that, when entropy evaluation cost is negligible, our method achieves superior convergence rates. Numerical experiments on high-dimensional inverse problems demonstrate substantial improvements in both computational efficiency and accuracy of EIG estimation.

Technology Category

• Machine Learning: Bayesian Learning
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Bayesian optimal experimental design provides a principled framework for selecting experimental settings that maximize obtained information. In this work, we focus on estimating the expected information gain in the setting where the differential entropy of the likelihood is either independent of the design or can be evaluated explicitly. This reduces the problem to maximum entropy estimation, alleviating several challenges inherent in expected information gain computation. Our study is motivated by large-scale inference problems, such as inverse problems, where the computational cost is dominated by expensive likelihood evaluations. We propose a computational approach in which the evidence density is approximated by a Monte Carlo or quasi-Monte Carlo surrogate, while the differential entropy is evaluated using standard methods without additional likelihood evaluations. We prove that this strategy achieves convergence rates that are comparable to, or better than, state-of-the-art methods for full expected information gain estimation, particularly when the cost of entropy evaluation is negligible. Moreover, our approach relies only on mild smoothness of the forward map and avoids stronger technical assumptions required in earlier work. We also present numerical experiments, which confirm our theoretical findings.

Problem

• Estimating expected information gain in Bayesian experimental design
• Reducing computational cost of likelihood evaluations in large-scale inference
• Developing efficient entropy approximation methods without additional likelihood evaluations

Innovation

• Monte Carlo surrogate approximates evidence density
• Standard methods evaluate differential entropy directly
• Method avoids expensive likelihood evaluations