The Two-Hump Problem: Bridging the Difficulty Gap in Mathematical Reinforcement Learning

2026-06-19

AI Summary

Mathematical reinforcement learning faces significant challenges due to an enormous search space, sparse rewards, and a bimodal distribution of problem instances—comprising only very easy or very hard samples—with a notable absence of intermediate-difficulty examples that hinders effective training. This work is the first to identify and address this bimodal difficulty gap by introducing a novel data generation method that populates the missing spectrum of problem hardness. The proposed approach integrates a hyper-action mechanism with a Transformer architecture to enhance both exploration and generalization capabilities. Evaluated on tasks such as the Andrews–Curtis conjecture, the method substantially outperforms existing baselines. Additionally, the study releases AC-19 and AC-1M, the first large-scale public datasets for this domain, containing 125,192 and 1,136,154 Andrews–Curtis trivial presentations spanning diverse difficulty levels, respectively.

Abstract

Mathematical search problems present a unique challenge for Reinforcement Learning (RL) due to vast search spaces and sparse rewards. In previous works, the Andrews-Curtis (AC) conjecture was established as an illustrative example of such problems. In this work, we identify a critical structural barrier in the AC landscape: a "Two-Hump" distribution, where problem instances are either trivially solvable or effectively impossible, with a scarcity of intermediate "hard-but-solvable" instances required for effective learning. We tackle this challenge through two primary avenues: novel data generation techniques to populate the difficulty gap, and significant algorithmic enhancements including the introduction of supermoves and Transformer-based architectures. We demonstrate substantial performance improvements over previous baselines, and release new comprehensive benchmark datasets including AC-19 (125,192 AC-trivial presentations of varying difficulty with length at most 19) and AC-1M (1,136,154 hard AC-trivial presentations of length at most 30), the first large-scale, publicly available datasets of this kind.

Problem

• Two-Hump Problem
• Mathematical Reinforcement Learning
• Andrews-Curtis conjecture
• sparse rewards
• difficulty gap

Innovation

• Two-Hump Problem
• supermoves
• Transformer-based RL
• difficulty gap
• mathematical reinforcement learning