OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learning

2026-04-20

AI Summary

This work addresses the challenge that large language models struggle to effectively explore novel reasoning paths beyond their initial latent space in reinforcement learning settings. To overcome this limitation, the authors propose a hybrid framework that integrates offline multi-teacher collaborative guidance with online reinforcement learning. The approach leverages trajectory-level offline supervision, entropy-aware exploration reward modulation, and uncertainty-driven reward modeling to steer the model toward efficient autonomous exploration in both mathematical and general reasoning tasks. Empirical results demonstrate that the method significantly outperforms existing baselines on mathematical reasoning benchmarks and exhibits strong out-of-distribution generalization capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Multiagent Systems: Multiagent Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent advancements in Reinforcement Learning with Verifiable Rewards (RLVR) have significantly improved Large Language Model (LLM) reasoning, yet models often struggle to explore novel trajectories beyond their initial latent space. While offline teacher guidance and entropy-driven strategies have been proposed to address this, they often lack deep integration or are constrained by the model's inherent capacity. In this paper, we propose OGER, a novel framework that unifies offline teacher guidance and online reinforcement learning through a specialized reward modeling lens. OGER employs multi-teacher collaborative training and constructs an auxiliary exploration reward that leverages both offline trajectories and the model's own entropy to incentivize autonomous exploration. Extensive experiments across mathematical and general reasoning benchmarks demonstrate that OGER significantly outperforms competitive baselines, achieving substantial gains in mathematical reasoning while maintaining robust generalization to out-of-domain tasks. We provide a comprehensive analysis of training dynamics and conduct detailed ablation studies to validate the effectiveness of our entropy-aware reward modulation. Our code is available at https://github.com/ecoli-hit/OGER.git.

Problem

• reinforcement learning
• exploration
• large language models
• offline guidance
• reasoning

Innovation

• Offline-Guided Exploration
• Hybrid Reinforcement Learning
• Entropy-Aware Reward
• Multi-Teacher Collaborative Training
• Auxiliary Exploration Reward