Learning Sketch Decompositions in Planning via Deep Reinforcement Learning

2024-12-11

arXiv.org

AI Summary

Identifying cross-task subgoal structures in long-horizon planning remains challenging due to the combinatorial complexity and lack of generalizable abstractions. Method: This paper proposes the first end-to-end sketch decomposition framework based on deep reinforcement learning (DRL), reformulating sketch discovery as a sequential decision-making task. It integrates IW(k) heuristic search, state-transition feature characterization, and policy network training to automatically learn interpretable and generalizable subgoal hierarchies—bypassing the scalability and expressivity limitations of prior feature-pooling and minimal SAT-solving approaches. Contribution/Results: Evaluated across multiple classical planning domains, the learned sketches enable efficient greedy IW(k) planning for original problems, yielding substantial improvements in cross-domain planning efficiency and zero-shot generalization. Crucially, the resulting subgoal structures retain explicit semantic interpretability, facilitating human understanding and downstream verification.

Technology Category

• Planning, Routing, and Scheduling: Learning for Planning and Scheduling
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Search and Optimization: Heuristic Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

In planning and reinforcement learning, the identification of common subgoal structures across problems is important when goals are to be achieved over long horizons. Recently, it has been shown that such structures can be expressed as feature-based rules, called sketches, over a number of classical planning domains. These sketches split problems into subproblems which then become solvable in low polynomial time by a greedy sequence of IW$(k)$ searches. Methods for learning sketches using feature pools and min-SAT solvers have been developed, yet they face two key limitations: scalability and expressivity. In this work, we address these limitations by formulating the problem of learning sketch decompositions as a deep reinforcement learning (DRL) task, where general policies are sought in a modified planning problem where the successor states of a state s are defined as those reachable from s through an IW$(k)$ search. The sketch decompositions obtained through this method are experimentally evaluated across various domains, and problems are regarded as solved by the decomposition when the goal is reached through a greedy sequence of IW$(k)$ searches. While our DRL approach for learning sketch decompositions does not yield interpretable sketches in the form of rules, we demonstrate that the resulting decompositions can often be understood in a crisp manner.

Problem

• Learning sketch decompositions via deep reinforcement learning
• Addressing scalability and expressivity limitations in planning
• Solving long-horizon goals through subproblem decomposition

Innovation

• Deep reinforcement learning for sketch decompositions
• Greedy IW(k) search sequence for subproblems
• Modified planning problem with successor states