Finding the Time to Think: Learning Planning Budgets in Real-Time RL

2026-06-24

AI Summary

This work addresses the challenge of real-time decision-making in reinforcement learning, where agents must act within limited and variable time budgets, as excessive planning can lead to paralysis by analysis. The authors propose a state-dependent planning budget mechanism that employs a lightweight gated policy network to dynamically allocate planning time at each step, enabling end-to-end optimization of “how long to think” conditioned on the current state. Integrated with a planning-based reinforcement learning agent, the method consistently outperforms fixed-budget and heuristic baselines across multiple real-time environments—including Pac-Man, Tetris, Snake, Speed Hex, and Speed Go—and demonstrates strong generalization under cross-GPU deployment, effectively balancing planning depth against response latency.

Abstract

Deliberating takes time. In real-time settings, that time is not free. Standard reinforcement learning (RL) sidesteps this as the environment waits indefinitely for the agent's decision. Instead, we study real-time RL environments where the environment progresses while waiting for the agent's action. Building on prior real-time formalizations, we introduce variable-delay real-time RL, where the agent chooses how long to deliberate at each decision point since the environment progresses. For the planning agents we use, the right delay is state-dependent, and naively planning how long to plan can paralyze the agent. We instead approach this setting by training a lightweight gating policy on top of a planner to select state-dependent planning budgets. Across real-time Pac-Man, Tetris, Snake, Speed Hex, and Speed Go, our gating policy outperforms fixed-budget and heuristic baselines, and transfers to a real-time setup where the environment and agent run on two different GPUs.

Problem

• real-time reinforcement learning
• planning budget
• variable delay
• deliberation time
• state-dependent planning

Innovation

• real-time reinforcement learning
• planning budget
• variable-delay RL
• gating policy
• state-dependent deliberation