Think Longer to Explore Deeper: Learn to Explore In-Context via Length-Incentivized Reinforcement Learning

2026-02-12

AI Summary

This work addresses the limitation of large language models during inference, where autoregressive generation suffers from exponentially decaying sampling probabilities over long reasoning trajectories, leading to a “shallow exploration trap” that hinders effective in-context exploration. To overcome this, the authors propose a reinforcement learning framework incorporating length-based rewards and redundancy penalties, which—building upon state coverage theory—encourages the generation of longer and more diverse reasoning paths. This approach marks the first integration of state coverage principles into in-context exploration. Evaluated on mainstream models including Qwen3 and Llama, the method significantly enhances exploratory capabilities, yielding average performance gains of 4.4% on in-domain tasks and 2.7% on cross-domain benchmarks, thereby improving both reasoning depth and generalization.

Technology Category

• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Achieving effective test-time scaling requires models to engage in In-Context Exploration -- the intrinsic ability to generate, verify, and refine multiple reasoning hypotheses within a single continuous context. Grounded in State Coverage theory, our analysis identifies a critical bottleneck to enabling this capability: while broader state coverage requires longer reasoning trajectories, the probability of sampling such sequences decays exponentially during autoregressive generation, a phenomenon we term the ``Shallow Exploration Trap''. To bridge this gap, we propose Length-Incentivized Exploration(\method). This simple yet effective recipe explicitly encourages models to explore more via a length-based reward coupled with a redundancy penalty, thereby maximizing state coverage in two-step manner. Comprehensive experiments across different models (Qwen3, Llama) demonstrate that \method effectively incentivize in-context exploration. As a result, our method achieves an average improvement of 4.4\% on in-domain tasks and a 2.7\% gain on out-of-domain benchmarks.

Problem

• In-Context Exploration
• State Coverage
• Shallow Exploration Trap
• Test-Time Scaling
• Reasoning Trajectories

Innovation

• In-Context Exploration
• Length-Incentivized Reinforcement Learning
• State Coverage
• Shallow Exploration Trap
• Autoregressive Generation