Hierarchical Reinforcement Learning for Sparse-Reward Search in Commutative Algebra

2026-06-22

AI Summary

This work addresses the challenge of constructing counterexamples in commutative algebra—such as those refuting Kalai’s algebraic Hirsch conjecture—where extremely sparse rewards hinder traditional search methods. To tackle this, we introduce hierarchical reinforcement learning (HRL) into the domain for the first time, proposing a novel HRL framework that integrates a constrained options mechanism with equivariant graph neural networks. The approach formulates the problem as a sparse-reward reinforcement learning task on graphs, leveraging the inherent hierarchy and symmetry of algebraic structures to learn temporally abstract, task-relevant policies that enable efficient exploration of the solution space. Experimental results demonstrate that our method substantially outperforms both classical reinforcement learning algorithms and greedy search across various degree settings, successfully enabling the efficient discovery of counterexamples to Kalai’s conjecture.

Abstract

Applying machine learning techniques to solving long-standing mathematical conjectures can be particularly challenging due to their extreme reward sparsity. As an illustrative example, we consider Kalai's algebraic Hirsch conjecture and recast the construction of its counterexamples as a sparse-reward reinforcement learning problem on graphs. We propose a constrained options-based HRL framework with an equivariant graph neural network policy, which allows us to learn useful temporal abstractions for this task. We evaluate our approach over a wide range of degrees and demonstrate that it consistently outperforms classical RL algorithms as well as greedy search. By exploiting the hierarchical structure of the problem, we effectively provide a first-of-its-kind application of HRL to a problem in commutative algebra.

Problem

• sparse-reward
• reinforcement learning
• commutative algebra
• mathematical conjectures
• counterexample construction

Innovation

• Hierarchical Reinforcement Learning
• Sparse-Reward RL
• Equivariant Graph Neural Network
• Commutative Algebra
• Constrained Options