Quantile of Means: A Bonus-Free Ensemble Method for Minimax Optimal Reinforcement Learning

2026-06-18

AI Summary

This work addresses the limitations of traditional count-based uncertainty estimation in reinforcement learning, which struggles to scale, and the lack of theoretical guarantees in existing ensemble methods. Focusing on finite-horizon Markov decision processes, the paper proposes a novel quantile-based ensemble exploration method that models the quantiles of the return distribution to drive effective exploration—without requiring reward bonuses or explicit state visitation counts. The approach provides, for the first time, rigorous theoretical support for ensemble-based exploration, achieving a minimax-optimal regret bound that depends on the variance of returns. This result establishes a solid theoretical foundation for efficient exploration in practical reinforcement learning settings.

Abstract

Optimal Reinforcement Learning (RL) algorithms typically rely on carefully constructed count-based uncertainty estimates to drive exploration. Although theoretically sound, such estimates are hard to compute in practical settings and therefore offer limited insight for designing exploration heuristics. Meanwhile, ensembling has emerged as a practical approach, but remains without theoretical justification. Building on a recent ensemble-based method for Multi-Armed Bandits, we propose a quantile-based ensemble method for finite-horizon Markov Decision Processes (MDPs). Our simple count-free approach achieves optimal variance-dependent regret bounds, providing theoretical grounding for ensemble-based exploration in RL.

Problem

• Reinforcement Learning
• Ensemble Methods
• Exploration
• Regret Bounds
• Theoretical Justification

Innovation

• quantile-based ensemble
• bonus-free exploration
• minimax optimal regret
• count-free RL
• variance-dependent bound