Graph Sparse Sampling: Breaking the Curse of the Horizon in Continuous MDP Planning

2026-07-06

AI Summary

This work addresses the exponential growth in computational complexity with planning horizon that plagues online planning in continuous Markov decision processes due to tree-based structures. To overcome this limitation, the authors propose the Graph-based Sparse Sampling (GSS) algorithm, which introduces a branching-free graph structure into online planning for continuous MDPs for the first time. GSS enables efficient allocation of computational resources by sharing sampled future trajectories across multiple candidate actions and integrating smooth backtracking with heuristic policies, while also supporting GPU-accelerated batch processing. Under conditions of trajectory overlap, regularity, and action coverage, theoretical analysis shows that GSS incurs only polynomial growth in performance error with respect to the planning horizon, thereby circumventing the exponential bottleneck inherent in traditional tree search. Empirical results demonstrate that GSS significantly outperforms existing tree-based planners in continuous control tasks, achieving near-optimal performance especially in long-horizon scenarios.

Abstract

Planning under uncertainty in continuous domains is essential for autonomous systems, yet computationally demanding. Tree-based search methods such as Monte Carlo Tree Search (MCTS) remain popular, but their branching structure can require sampling budgets that grow exponentially with lookahead depth in the worst case. From a tree perspective, continuous state or action spaces become especially challenging, since the planner must decide where to search in an infinite branching hierarchy. We propose Graph Sparse Sampling (GSS), an online planning algorithm that shares sampled futures across many candidate decisions, rather than sampling separate successors for each candidate action. This branch-free graph exposes large GPU-friendly batches, while using heuristics to focus computation. We prove finite-sample performance guarantees for GSS covering full-rank or low-rank generative simulators via smoothed backups, and discrete or sampled continuous action spaces. Under suitable overlap, regularity, and action-coverage conditions, these bounds have polynomial dependence on the planning horizon, formalizing when shared futures can avoid the exponential horizon dependence of tree-shaped sparse sampling. We demonstrate continuous-control simulations where GSS substantially outperforms tree-based planners on long horizons or achieves near-optimal performance, supporting no-branching graph planning as a complementary design principle for online control.

Problem

• continuous MDP planning
• curse of the horizon
• sparse sampling
• online planning
• branching complexity

Innovation

• Graph Sparse Sampling
• online planning
• continuous MDP
• shared futures
• horizon curse