Stagewise Reinforcement Learning and the Geometry of the Regret Landscape

2026-01-12

AI Summary

This study investigates the relationship between policy evolution and the geometric structure of the regret function in deep reinforcement learning, aiming to uncover the origins of staged learning phenomena. By extending singular learning theory to the reinforcement learning framework, the work introduces the local learning coefficient (LLC) to characterize the concentration behavior of the generalized posterior over the policy space, thereby establishing a theoretical link between the geometry of the regret function and policy phase transitions. The analysis reveals that LLC can detect internal algorithmic evolution during performance plateaus. In grid-world environments, an “opposing staircase” phenomenon is observed: sharp drops in regret coincide with increases in LLC, empirically validating a staged transition mechanism from simple, high-regret policies to complex, low-regret ones.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Online Learning & Bandits
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Singular learning theory characterizes Bayesian learning as an evolving tradeoff between accuracy and complexity, with transitions between qualitatively different solutions as sample size increases. We extend this theory to deep reinforcement learning, proving that the concentration of the generalized posterior over policies is governed by the local learning coefficient (LLC), an invariant of the geometry of the regret function. This theory predicts that Bayesian phase transitions in reinforcement learning should proceed from simple policies with high regret to complex policies with low regret. We verify this prediction empirically in a gridworld environment exhibiting stagewise policy development: phase transitions over SGD training manifest as"opposing staircases"where regret decreases sharply while the LLC increases. Notably, the LLC detects phase transitions even when estimated on a subset of states where the policies appear identical in terms of regret, suggesting it captures changes in the underlying algorithm rather than just performance.

Problem

• reinforcement learning
• phase transitions
• regret landscape
• local learning coefficient
• Bayesian learning

Innovation

• Local Learning Coefficient
• Singular Learning Theory
• Bayesian Phase Transitions
• Regret Landscape Geometry
• Stagewise Reinforcement Learning