Learning to Reason with Curriculum II: Compositional Generalization

2026-06-26

AI Summary

This work addresses the poor generalization and low sample efficiency of intelligent systems in long-sequence computational tasks by proposing a self-curriculum recursive decomposition method. The approach recursively decomposes complex tasks into shorter subproblems, which are learned individually and then composed to solve the original task. Integrating finite automaton simulation, supervised fine-tuning (SFT), and reinforcement learning with verifiable rewards (RLVR), this method provides the first theoretical guarantee that curriculum learning can reduce the sample complexity of supervised learning from Ω(T) to 2^O(√log T). It also substantially relaxes the coverage requirement on reference models in reinforcement learning—requiring only local supervision over blocks of length B ≪ T—to enable efficient learning. Empirically, the method achieves scalable solutions with sub-polynomial sample complexity on simulated automaton tasks.

Abstract

Compositional generalization, the ability to solve complex problems by combining solutions to simpler sub-problems, is a fundamental capability of both natural and artificial intelligence, and a key mechanism underlying chain-of-thought reasoning. However, the theoretical underpinnings of compositional generalization remain poorly understood: when and why does decomposing a problem into parts yield more efficient learning than solving it directly? We study this question through the canonical problem of learning to simulate semiautomata (predicting the outcome of $T$ steps of sequential computation), a model that captures state tracking, regular language recognition, and modular arithmetic. We show that an autocurriculum-based approach building on Part I of this series, recursively decomposing longer sequences into shorter sub-problems, learning to solve them, and composing the solutions, achieves dramatically better statistical complexity than direct methods. (i) For a setting inspired by supervised fine-tuning (SFT) where the learner receives interactive feedback on intermediate states of the computation, curriculum facilitates learning from only $2^{\mathcal{O}(\sqrt{\log T})}$ tokens of supervision; i.e., subpolynomial in the sequence length $T$, overcoming the $Ω(T)$ token barrier required by direct simulation. (ii) For a setting inspired by reinforcement learning with verifiable rewards (RLVR), where the learner improves a pre-trained reference model using an outcome verifier, we show that curriculum reduces the requirement on the reference model from coverage at the full sequence length $T$ to coverage at a shorter block length $B \ll T$, an exponentially weaker condition.

Problem

• compositional generalization
• semiautomata
• curriculum learning
• statistical complexity
• sequence decomposition

Innovation

• compositional generalization
• autocurriculum
• semiautomata simulation
• statistical complexity
• chain-of-thought reasoning