Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMs

2026-06-16

AI Summary

This work addresses the “autoregressive curse”—a phenomenon in large language models where minor early perturbations during long-horizon logical reasoning trigger irreversible cascading errors—by introducing the E³RL method. E³RL uniquely leverages endogenous local autoregressive cross-entropy as both a measure of cognitive uncertainty and an intrinsic reward signal, integrated with segment-level dynamic thresholds, an advantage allocation mechanism, and KV cache reuse to precisely excise and self-repair logical flaws without external supervision. Experiments demonstrate that E³RL enables 4B and 8B models to surpass prior state-of-the-art performance on mathematical reasoning benchmarks such as AIME by 5.35% and 6.51%, respectively, substantially improving exploration and sample efficiency in long-sequence reasoning while maintaining linear memory overhead.

Abstract

Although reinforcement learning (RL) has expanded the cognitive boundaries of large language models (LLMs), it often remains vulnerable to the autoregressive curse in long-horizon logical reasoning: small epistemic perturbations introduced early in generation can propagate irreversibly along the Markov decision process flow, triggering cascading failures that drive the reasoning trajectory toward collapse. To overcome this autoregressive cascade, in which a single early mistake can compromise all subsequent reasoning steps, we propose dynamic epistemic entropy orchestrated erasable reinforcement learning ($\text{E}^3\text{RL}$). $\text{E}^3\text{RL}$ eliminates reliance on external signals by grounding the model's endogenous local autoregressive cross-entropy as an intrinsic coordinate of epistemic uncertainty. By introducing segment-level adaptive dynamic thresholds and advantage allocation, $\text{E}^3\text{RL}$ enables the model to precisely excise localized logical defects while reusing historical key-value (KV) cache streams, thereby endowing the reasoning process with a self-healing capability. We train $\text{E}^3\text{RL}$ on the DeepMath-103k dataset. Experimental results show that $\text{E}^3\text{RL}$ reshapes the exploration efficiency of long-sequence reasoning and improves sample efficiency while maintaining linear memory overhead. On mathematical reasoning benchmarks such as AIME, $\text{E}^3\text{RL}$ achieves substantial performance gains, with the 4B and 8B parameter models surpassing previous state-of-the-art (SOTA) results by 5.349\% and 6.514\%, respectively. These findings suggest that $\text{E}^3\text{RL}$ shatters the autoregressive curse in long-sequence reasoning and establishes a theoretical and systems-level foundation for the next generation of self-healing artificial general intelligence (AGI).

Problem

• autoregressive curse
• long-horizon reasoning
• cascading failures
• epistemic perturbations
• logical reasoning

Innovation

• erasable reinforcement learning
• epistemic entropy
• autoregressive curse
• self-healing reasoning
• KV cache reuse