Variance Driven Exploration: A Provable and Efficient Methodology for Pure Exploration in Highly Stochastic Environments

2026-08-22

AI Summary

本文提出了一种名为VarDE的方法，通过最小化最终决策的不确定性来解决高度随机环境中的纯探索问题，并在多个核心问题上展示了其优越性。

Abstract

We propose Variance Driven Exploration (VarDE), a principled approach for pure exploration in highly stochastic environments, where the exploration process is dominated by stochastic variance. VarDE is built on a fundamental principle: sampling effort should be allocated to minimize the uncertainty of the final decision. We formalize the uncertainty of the final decision through a smooth decision function and derive allocation rules that explicitly capture how stochastic noise in individual components affects the reliability of the final output. We apply this methodology to three core problems of pure exploration -- Best Arm Identification (BAI), Monte Carlo Tree Search (MCTS), and Best-Policy Identification (BPI) -- with theoretical guarantees on variance decay and simple regret. Empirically, we demonstrate consistent and significant improvements of VarDE over existing methods, with especially strong gains in highly stochastic environments.

Problem

• Variance Driven Exploration
• stochastic environments
• pure exploration

Innovation

• Variance Driven Exploration
• stochastic environments
• sampling effort allocation
• uncertainty minimization
• pure exploration