When in Doubt, Plan It Out: Committed Small Language Model Deliberation for Reactive Reinforcement Learning

2026-06-15

AI Summary

Reinforcement learning policies often underperform in unfamiliar environments due to their lack of explicit reasoning capabilities. To address this limitation, this work proposes the PACT architecture, which introduces a small language model (SLM, 2B parameters) into an asynchronous, commitment-based planning framework for the first time. In PACT, a slow SLM-based planner operates in the background to generate and simulate action plans, while a fast reinforcement learning policy handles real-time execution; the two components operate in a decoupled yet coordinated manner. This design enhances system robustness and adaptability without requiring modifications to the original policy. Evaluated on three progressively challenging FrozenLake tasks, PACT consistently outperforms baseline methods, demonstrating the effectiveness of integrating reactive control with symbolic planning.

Abstract

Reinforcement Learning (RL) policies often degrade in unfamiliar environments because they lack explicit deliberation. We propose Plan, Align, Commit, Think (PACT), a hybrid architecture that combines a fast, reactive RL policy with a slow, deliberative Small Language Model (SLM) planner. PACT invokes the SLM asynchronously to generate and validate candidate action plans. Once a plan is verified through simulation as safe, feasible, and complete, it is executed directly, bypassing the RL policy without retraining or modifying it. Evaluated on three FrozenLake configurations of increasing difficulty, PACT outperforms all baselines while relying on a 2B-parameter SLM backbone, suggesting that deliberative planning and reactive execution are more powerful in concert than either is alone in these settings.

Problem

• Reinforcement Learning
• deliberation
• planning
• out-of-distribution generalization
• reactive policy

Innovation

• Small Language Model
• Reinforcement Learning
• Deliberative Planning
• Hybrid Architecture
• Asynchronous Planning