Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents

2026-01-31

AI Summary

Current large language models and deep reinforcement learning systems struggle to achieve human-like efficient planning and cross-task generalization through abstraction, often relying on manually provided abstract knowledge. This work proposes TheoryCoder-2, the first framework that enables automatic abstraction learning within a theory-driven reinforcement learning paradigm. By leveraging the in-context learning capabilities of large language models, TheoryCoder-2 actively synthesizes reusable abstractions from experience and integrates them into hierarchical planning. The method requires no human-specified abstractions and demonstrates substantial improvements in sample efficiency and task success rates across diverse environments—including BabyAI, Minihack, and VGDL—solving several complex tasks that baseline approaches fail to complete, with only minimal human prompting.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Humans learn abstractions and use them to plan efficiently to quickly generalize across tasks -- an ability that remains challenging for state-of-the-art large language model (LLM) agents and deep reinforcement learning (RL) systems. Inspired by the cognitive science of how people form abstractions and intuitive theories of their world knowledge, Theory-Based RL (TBRL) systems, such as TheoryCoder, exhibit strong generalization through effective use of abstractions. However, they heavily rely on human-provided abstractions and sidestep the abstraction-learning problem. We introduce TheoryCoder-2, a new TBRL agent that leverages LLMs'in-context learning ability to actively learn reusable abstractions rather than relying on hand-specified ones, by synthesizing abstractions from experience and integrating them into a hierarchical planning process. We conduct experiments on diverse environments, including BabyAI, Minihack and VGDL games like Sokoban. We find that TheoryCoder-2 is significantly more sample-efficient than baseline LLM agents augmented with classical planning domain construction, reasoning-based planning, and prior program-synthesis agents such as WorldCoder. TheoryCoder-2 is able to solve complex tasks that the baselines fail, while only requiring minimal human prompts, unlike prior TBRL systems.

Problem

• abstraction learning
• hierarchical planning
• program synthesis
• generalization
• large language models

Innovation

• abstraction learning
• hierarchical planning
• in-context learning
• program synthesis
• Theory-Based RL