Learning to Search and Searching to Learn for Generalization in Planning

2026-05-25

AI Summary

This work addresses the poor generalization and inefficient exploration of deep reinforcement learning (DRL) in sparse-reward combinatorial planning tasks by proposing a self-improving weighted A* (WA*) learning framework. Integrating relational graph neural networks with symbolic state representations, the method employs Q-learning to self-supervise the update of its heuristic function, forming a closed loop with the search process—without requiring expert demonstrations or counterfactual relabeling. The approach achieves, for the first time, strong zero-shot generalization: it substantially outperforms conventional DRL methods on benchmarks including Sokoban, PushWorld, The Witness, and IPC-2023, and successfully transfers policies trained on small-scale Blocksworld instances (30 blocks) to solve vastly larger ones (488 blocks) without further training.

Abstract

Combinatorial generalization remains a central challenge in Deep Reinforcement Learning (DRL). Classical planning provides a simple yet challenging setting to study this problem through explicit relational descriptions, without requiring learning from perception. In sparse-reward domains, standard RL exploration via real-time search is ineffective, and learning-based planning methods often rely on expert demonstrations, hindsight relabeling, or random walks from the goal state. In contrast, planners rely on best-first search methods such as $\mathrm{A}^\star$ to solve problems from scratch. We propose a self-improving $\mathrm{WA}^\star$ learning framework in combination with a value heuristic represented by a Relational Graph Neural Network: the heuristic guides search, and the resulting search data updates the heuristic via $Q$-learning. This loop yields heuristics that can function as general policies and solve new instances even without search, where DRL otherwise fails, as we show on puzzles such as Sokoban, PushWorld, The Witness, and the 2023 International Planning Competition benchmarks. Notably, we demonstrate strong zero-shot generalization: For example, heuristics trained on Blocksworld instances with fewer than 30 blocks successfully solve instances with 488 blocks without search.

Problem

• combinatorial generalization
• deep reinforcement learning
• sparse-reward domains
• planning
• zero-shot generalization

Innovation

• Weighted A*
• Relational Graph Neural Network
• Zero-shot Generalization
• Self-improving Planning
• Combinatorial Generalization